Sep 07 22:30:12.269051 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 07 22:30:12.269060 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 07 22:30:12.269067 ip-10-0-143-210 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 07 22:30:12.269218 ip-10-0-143-210 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 07 22:30:23.635092 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 07 22:30:23.635112 ip-10-0-143-210 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 72d640fe50aa4847b381adb90b7e39bb --
Sep 07 22:32:00.044386 ip-10-0-143-210 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 22:32:00.462708 ip-10-0-143-210 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:00.462708 ip-10-0-143-210 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 22:32:00.462708 ip-10-0-143-210 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:00.462708 ip-10-0-143-210 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 22:32:00.462708 ip-10-0-143-210 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:00.463342 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.462772    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 22:32:00.466695 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466679    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.466695 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466694    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466698    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466701    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466704    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466707    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466710    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466713    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466715    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466718    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466720    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466723    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466727    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466731    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466735    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466737    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466740    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466743    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466747    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466750    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.466757 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466753    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466756    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466759    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466762    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466764    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466767    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466770    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466773    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466776    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466778    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466781    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466783    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466785    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466788    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466791    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466794    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466796    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466799    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466802    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466804    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.467220 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466807    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466809    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466812    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466814    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466817    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466820    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466823    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466825    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466828    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466830    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466833    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466835    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466838    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466840    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466845    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466847    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466850    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466853    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466856    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466858    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.467705 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466861    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466864    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466866    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466868    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466871    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466873    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466876    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466878    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466882    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466885    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466888    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466891    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466893    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466896    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466898    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466901    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466904    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466906    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466910    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.468218 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466913    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466916    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466918    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466921    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466923    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466925    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466928    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466931    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466934    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466937    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466939    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466942    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466944    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466947    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466949    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466952    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466954    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466957    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466959    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.468676 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.466961    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.469468 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469454    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.469468 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469466    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469470    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469474    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469477    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469479    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469482    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469485    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469488    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469490    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469494    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469497    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469501    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469505    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469509    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469512    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469515    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469518    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469521    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469524    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.469525 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469527    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469530    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469533    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469536    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469539    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469541    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469544    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469547    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469550    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469552    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469555    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469558    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469560    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469563    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469565    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469568    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469570    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469573    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469576    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469578    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.469991 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469580    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469583    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469585    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469588    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469590    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469593    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469596    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469599    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469601    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469604    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469606    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469608    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469611    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469613    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469617    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469621    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469624    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469626    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469629    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469632    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.470575 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469634    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469637    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469639    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469642    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469644    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469647    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469649    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469652    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469654    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469657    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469659    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469662    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469664    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469667    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469669    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469672    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469674    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469677    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469681    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.471356 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469684    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469687    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469689    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469692    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469694    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469697    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469699    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469703    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469706    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469708    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469711    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469713    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469716    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469718    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469721    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469723    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469726    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469729    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469731    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.471852 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.469734    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470095    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470100    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470103    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470106    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470109    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470112    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470115    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470118    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470121    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470137    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470140    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470143    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470147    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470149    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470152    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470155    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470158    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470161    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470164    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.472347 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470168    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470171    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470174    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470176    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470180    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470182    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470185    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470188    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470190    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470193    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470196    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470198    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470201    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470204    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470206    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470209    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470212    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470216    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470219    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.472837 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470223    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470225    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470228    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470231    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470234    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470238    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470242    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470245    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470248    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470250    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470252    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470255    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470258    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470261    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470264    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470266    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470269    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470272    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470274    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.473342 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470279    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470282    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470285    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470288    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470291    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470293    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470296    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470298    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470301    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470303    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470306    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470308    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470311    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470314    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470316    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470318    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470321    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470324    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470327    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470330    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.473806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470332    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470371    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470562    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470568    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470572    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470577    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470582    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470588    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470593    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470598    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470602    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470606    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470610    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470614    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470619    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470628    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470632    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470636    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470640    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470645    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.474296 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470650    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.470654    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470769    2550 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470825    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470842    2550 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470849    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470856    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470863    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470870    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470877    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470887    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470892    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470897    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470902    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470908    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470913    2550 flags.go:64] FLAG: --cgroup-root=""
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470918    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470922    2550 flags.go:64] FLAG: --client-ca-file=""
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470927    2550 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470936    2550 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470942    2550 flags.go:64] FLAG: --cluster-domain=""
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.470985    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471009    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471017    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 22:32:00.474806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471024    2550 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471051    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471056    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471062    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471067    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471079    2550 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471084    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471089    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471095    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471100    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471106    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471111    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471116    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471120    2550 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471146    2550 flags.go:64] FLAG: --enable-server="true"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471151    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471161    2550 flags.go:64] FLAG: --event-burst="100"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471166    2550 flags.go:64] FLAG: --event-qps="50"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471170    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471175    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471181    2550 flags.go:64] FLAG: --eviction-hard=""
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471187    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471196    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471201    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471206    2550 flags.go:64] FLAG: --eviction-soft=""
Sep 07 22:32:00.475390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471211    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471216    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471221    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471225    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471231    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471237    2550 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471247    2550 flags.go:64] FLAG: --feature-gates=""
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471253    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471257    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471262    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471268    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471273    2550 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471278    2550 flags.go:64] FLAG: --help="false"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471282    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471303    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471308    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471312    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471328    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471335    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471340    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471344    2550 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471347    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471367    2550 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 22:32:00.475983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471371    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471388    2550 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471487    2550 flags.go:64] FLAG: --kube-reserved=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471498    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471504    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471509    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471516    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471519    2550 flags.go:64] FLAG: --lock-file=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471522    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471525    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471528    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471534    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471536    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471539    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471542    2550 flags.go:64] FLAG: --logging-format="text"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471545    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471549    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471551    2550 flags.go:64] FLAG: --manifest-url=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471554    2550 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471559    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471563    2550 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471567    2550 flags.go:64] FLAG: --max-pods="110"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471570    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471573    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471576    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 22:32:00.476590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471579    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471582    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471585    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471589    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471597    2550 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471600    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471604    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471607    2550 flags.go:64] FLAG: --pod-cidr=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471610    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471616    2550 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471619    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471622    2550 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471625    2550 flags.go:64] FLAG: --port="10250"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471628    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471632    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05e39060ceb8fb50f"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471635    2550 flags.go:64] FLAG: --qos-reserved=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471638    2550 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471641    2550 flags.go:64] FLAG: --register-node="true"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471644    2550 flags.go:64] FLAG: --register-with-taints=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471647    2550 flags.go:64] FLAG: --registry-burst="10"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471650    2550 flags.go:64] FLAG: --registry-qps="5"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471653    2550 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471656    2550 flags.go:64] FLAG: --reserved-memory=""
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471659    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471663    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 22:32:00.477222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471666    2550 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471669    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471672    2550 flags.go:64] FLAG: --runonce="false"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471675    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471678    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471681    2550 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471684    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471687    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471690    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471693    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471696    2550 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471698    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471701    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471705    2550 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471708    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471711    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471715    2550 flags.go:64] FLAG: --system-cgroups=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471718    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471724    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471726    2550 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471729    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471735    2550 flags.go:64] FLAG: --tls-min-version=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471738    2550 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471741    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471744    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 22:32:00.477853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471747    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471750    2550 flags.go:64] FLAG: --v="2"
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471754    2550 flags.go:64] FLAG: --version="false"
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471758    2550 flags.go:64] FLAG: --vmodule=""
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471761    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.471765    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471875    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471879    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471883    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471885    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471888    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471891    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471896    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471900    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471903    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471906    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471910    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471913    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471915    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471918    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471921    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.478508 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471924    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471927    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471929    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471932    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471934    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471937    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471939    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471943    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471945    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471948    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471950    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471953    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471957    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471960    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471963    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471966    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471968    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471971    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471973    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.479007 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471976    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471978    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471981    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471983    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471986    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471989    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471993    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.471996    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472004    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472007    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472010    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472012    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472015    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472018    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472021    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472023    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472025    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472028    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472031    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472033    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.479482 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472036    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472038    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472041    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472044    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472046    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472049    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472051    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472054    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472056    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472059    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472062    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472064    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472075    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472078    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472081    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472083    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472086    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472088    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472091    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.479978 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472093    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472096    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472098    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472101    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472104    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472107    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472109    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472112    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472114    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472117    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472119    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472122    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472138    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472141    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472143    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472146    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472149    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472151    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472154    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472156    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.480503 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472159    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472162    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472164    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472167    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472170    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.472172    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.472182    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.478768    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.478781    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.478793    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.478802    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478852    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478857    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478860    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.480983 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478863    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478867    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478870    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478873    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478875    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478878    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478881    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478885    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478889    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478892    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478894    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478897    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478900    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478902    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478905    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478908    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478910    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478913    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478915    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.481341 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478918    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478921    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478923    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478926    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478928    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478931    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478934    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478937    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478939    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478942    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478944    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478947    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478949    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478952    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478954    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478962    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478964    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478967    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478969    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478972    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.481875 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478974    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478977    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478979    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478982    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478985    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478987    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478989    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478992    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478995    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.478997    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479000    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479002    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479006    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479009    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479013    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479016    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479019    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479022    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479025    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479027    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.482395 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479030    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479032    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479035    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479037    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479040    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479042    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479045    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479048    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479051    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479054    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479057    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479059    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479062    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479064    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479067    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479069    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479072    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479074    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479077    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479079    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.482984 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479082    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479085    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479088    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479090    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479092    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479096    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479099    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479102    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479105    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479108    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479111    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479113    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479116    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479119    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479122    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479141    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.483572 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479144    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.479149    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479253    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479257    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479261    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479267    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479271    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479275    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479279    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479284    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479288    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479292    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479295    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479298    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479301    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479304    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:00.484399 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479307    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479309    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479312    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479314    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479317    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479320    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479322    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479324    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479327    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479329    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479332    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479334    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479337    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479339    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479342    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479345    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479347    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479349    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479352    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:00.484851 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479355    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479357    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479359    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479363    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479365    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479368    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479370    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479373    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479375    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479378    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479381    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479385    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479388    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479391    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479394    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479396    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479399    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479401    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479404    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479406    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:00.485373 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479409    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479411    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479413    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479416    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479418    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479421    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479423    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479426    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479428    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479431    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479433    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479436    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479438    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479440    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479443    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479445    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479448    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479451    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479453    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:00.485934 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479456    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479459    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479461    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479464    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479466    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479470    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479473    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479476    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479479    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479481    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479484    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479487    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479489    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479492    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479495    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479498    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479500    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479502    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479505    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479507    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:00.486433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479510    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479512    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479516    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479522    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479524    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479527    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:00.479529    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.479534    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.479620    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.481495    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.482330    2550 server.go:1019] "Starting client certificate rotation"
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.482444    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.482494    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:32:00.486947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.484717    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:32:00.487458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.485775    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:32:00.499253 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.499235    2550 log.go:25] "Validated CRI v1 runtime API"
Sep 07 22:32:00.503522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.503507    2550 log.go:25] "Validated CRI v1 image API"
Sep 07 22:32:00.505023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.505009    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 22:32:00.509802 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.509783    2550 fs.go:135] Filesystem UUIDs: map[5a129402-2d19-49f9-b39d-7a6edc511ea9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ae0bd86-88e8-43c8-8a52-4f24ac39a6b6:/dev/nvme0n1p4]
Sep 07 22:32:00.509876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.509801    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 22:32:00.511408 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.511392    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:32:00.514675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.514573    2550 manager.go:216] Machine: {Timestamp:2026-09-07 22:32:00.51327518 +0000 UTC m=+0.355543438 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101252 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec236ab0965991e1c9537eaa341e19d2 SystemUUID:ec236ab0-9659-91e1-c953-7eaa341e19d2 BootID:72d640fe-50aa-4847-b381-adb90b7e39bb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:04:c2:be:23:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:04:c2:be:23:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:63:67:97:5a:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 22:32:00.514675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.514669    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 22:32:00.514779 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.514739    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 22:32:00.517769 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.517748    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 22:32:00.517884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.517771    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 22:32:00.519018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.519008    2550 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 22:32:00.519057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.519020    2550 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 22:32:00.519057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.519028    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:32:00.520209 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.520199    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:32:00.521411 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.521402    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 22:32:00.524992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.524981    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:32:00.525104 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525095    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 22:32:00.525145 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525136    2550 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 22:32:00.525217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525149    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 22:32:00.525217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525173    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 22:32:00.525217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525187    2550 kubelet.go:398] "Adding apiserver pod source"
Sep 07 22:32:00.525217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525197    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 22:32:00.525654 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525635    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:32:00.525654 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.525655    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:32:00.527538 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.527524    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 22:32:00.529873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.529859    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 22:32:00.529937 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.529889    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 22:32:00.530075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530063    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 22:32:00.530107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530081    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 22:32:00.530107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530087    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 22:32:00.530107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530092    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 22:32:00.530107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530100    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 22:32:00.530107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530106    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 22:32:00.530641 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530632    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 22:32:00.530641 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530641    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 22:32:00.530696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530648    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 22:32:00.530696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530653    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 22:32:00.530696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530663    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 22:32:00.530696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.530671    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 22:32:00.531393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.531385    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 22:32:00.531393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.531394    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 22:32:00.534896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.534880    2550 server.go:1268] "Started kubelet"
Sep 07 22:32:00.534992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.534940    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 22:32:00.535379 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.535227    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 22:32:00.535562 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.535544    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 22:32:00.535679 ip-10-0-143-210 systemd[1]: Started Kubernetes Kubelet.
Sep 07 22:32:00.536925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.536902    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49mg4"
Sep 07 22:32:00.538035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.537777    2550 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 22:32:00.539363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.538350    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 22:32:00.539363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.538780    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 22:32:00.539363 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.538916    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 22:32:00.539363 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.538947    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 22:32:00.540634 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.540600    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 22:32:00.542751 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.542733    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 22:32:00.542825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.542753    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 22:32:00.542882 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.542828    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:00.543401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543382    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 22:32:00.543491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543413    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 22:32:00.543491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543391    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 22:32:00.543599 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543521    2550 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 22:32:00.543599 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543530    2550 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 22:32:00.543716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543700    2550 factory.go:153] Registering CRI-O factory
Sep 07 22:32:00.543818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543740    2550 factory.go:223] Registration of the crio container factory successfully
Sep 07 22:32:00.543818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543786    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 22:32:00.543818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543795    2550 factory.go:55] Registering systemd factory
Sep 07 22:32:00.543818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543802    2550 factory.go:223] Registration of the systemd container factory successfully
Sep 07 22:32:00.543818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543820    2550 factory.go:103] Registering Raw factory
Sep 07 22:32:00.544060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.543835    2550 manager.go:1196] Started watching for new ooms in manager
Sep 07 22:32:00.544137 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.544111    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49mg4"
Sep 07 22:32:00.544220 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.544208    2550 manager.go:349] Starting recovery of all containers
Sep 07 22:32:00.549142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.549103    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:00.552165 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.552142    2550 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.554961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.554945    2550 manager.go:354] Recovery completed
Sep 07 22:32:00.560675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.560662    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.562606 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.562590    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.562673 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.562621    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.562673 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.562634    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.563217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563113    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 22:32:00.563217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563138    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 22:32:00.563217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563155    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:32:00.563217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563219    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.563910 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563894    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.563978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563921    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.563978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.563931    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.564289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.564277    2550 policy_none.go:49] "None policy: Start"
Sep 07 22:32:00.564421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.564297    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 22:32:00.564452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.564432    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 22:32:00.643323 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.643292    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:00.707760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.707741    2550 policy_none.go:47] "Start"
Sep 07 22:32:00.742626 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.742612    2550 manager.go:337] "Starting Device Plugin manager"
Sep 07 22:32:00.742704 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.742650    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 22:32:00.742704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.742662    2550 server.go:87] "Starting device plugin registration server"
Sep 07 22:32:00.742940 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.742928    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 22:32:00.743009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.742940    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 22:32:00.743056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.743025    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 22:32:00.743146 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.743111    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 22:32:00.743146 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.743141    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 22:32:00.743663 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.743646    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 22:32:00.743741 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.743687    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:00.788112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.788086    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 22:32:00.789205 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.789182    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 22:32:00.789275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.789235    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 22:32:00.789275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.789255    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 22:32:00.789368 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.789314    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 22:32:00.791207 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.791188    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:00.843874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.843846    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.844944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.844930    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.845010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.844958    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.845010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.844969    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.845010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.844988    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.852116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.852102    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.852164 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.852123    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-210.ec2.internal\": node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:00.866061 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.866041    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:00.889592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.889573    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"]
Sep 07 22:32:00.889648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.889638    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.890985 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.890972    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.891046 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.890999    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.891046 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.891010    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.893246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893234    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.893373 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893358    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.893423 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893387    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.893962 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893939    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.893962 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893960    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.894084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.893974    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.894084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.894016    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.894084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.894036    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.894084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.894047    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.896022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.896006    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.896111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.896029    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:00.896713 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.896698    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:00.896790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.896720    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:00.896790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.896733    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:00.911631 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.911613    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.914880 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.914864    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.945324 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.945309    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.945398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.945331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.945398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:00.945348    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f2c00fd5112014ad23844afadb2fd55-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"2f2c00fd5112014ad23844afadb2fd55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:00.966525 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:00.966507    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.046028 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.045987    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f2c00fd5112014ad23844afadb2fd55-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"2f2c00fd5112014ad23844afadb2fd55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.046028 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.046011    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.046111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.046028    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.046111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.046089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.046111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.046097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f669d2825f8c0e6d80ec2d9fbc641aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"f669d2825f8c0e6d80ec2d9fbc641aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.046228 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.046096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f2c00fd5112014ad23844afadb2fd55-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"2f2c00fd5112014ad23844afadb2fd55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.067102 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:01.067078    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.167635 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:01.167608    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.214780 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.214760    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.217870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.217856    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.267726 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:01.267703    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.368738 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:01.368685    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.469268 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:01.469236    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found"
Sep 07 22:32:01.481829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.481803    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 22:32:01.481955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.481937    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 22:32:01.482017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.481954    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 22:32:01.531139 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.531117    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:01.543348 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.543331    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 22:32:01.543434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.543337    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.546192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.546159    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 22:27:00 +0000 UTC" deadline="2028-04-03 07:27:34.053230233 +0000 UTC"
Sep 07 22:32:01.546241 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.546192    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13760h55m32.50704083s"
Sep 07 22:32:01.553887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.553860    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:32:01.555643 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.555627    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:32:01.556188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.556159    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal"
Sep 07 22:32:01.564564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.564547    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:32:01.572317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.572296    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4hcwt"
Sep 07 22:32:01.579364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.579349    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4hcwt"
Sep 07 22:32:01.718167 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.718147    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:01.792952 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:01.792916    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf669d2825f8c0e6d80ec2d9fbc641aea.slice/crio-edead60628e46ace9271bb2cc041e4ce596f32f7e9f956b6d3611d99fdb1bcc9 WatchSource:0}: Error finding container edead60628e46ace9271bb2cc041e4ce596f32f7e9f956b6d3611d99fdb1bcc9: Status 404 returned error can't find the container with id edead60628e46ace9271bb2cc041e4ce596f32f7e9f956b6d3611d99fdb1bcc9
Sep 07 22:32:01.797234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:01.797213    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:32:01.802205 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:01.802184    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2c00fd5112014ad23844afadb2fd55.slice/crio-5366aaa215bfe977346b545c39edccee3226763943119251e13cfc2c47f9b6fc WatchSource:0}: Error finding container 5366aaa215bfe977346b545c39edccee3226763943119251e13cfc2c47f9b6fc: Status 404 returned error can't find the container with id 5366aaa215bfe977346b545c39edccee3226763943119251e13cfc2c47f9b6fc
Sep 07 22:32:02.512155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.512115    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:02.526204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.526179    2550 apiserver.go:52] "Watching apiserver"
Sep 07 22:32:02.533785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.533757    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 22:32:02.536117 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.536092    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-chkft","openshift-ovn-kubernetes/ovnkube-node-kcfs8","kube-system/global-pull-secret-syncer-c7772","kube-system/konnectivity-agent-9b7wn","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5v5kv","openshift-image-registry/node-ca-mwfht","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","openshift-multus/network-metrics-daemon-hds5x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj","openshift-dns/node-resolver-jz776","openshift-multus/multus-9nfq6","openshift-multus/multus-additional-cni-plugins-mcb92","openshift-network-diagnostics/network-check-target-g2k4w"]
Sep 07 22:32:02.539090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.539044    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.541260 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.541229    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.542323 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.542220    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 22:32:02.542323 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.542296    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.542663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.542487    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.542767 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.542707    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fw4kc\""
Sep 07 22:32:02.544463 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.543776    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.545245 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.544905    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.545245 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.545053    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 22:32:02.545245 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.545139    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.545245 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.545184    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9cms\""
Sep 07 22:32:02.545723 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.545701    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 22:32:02.546201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.546142    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 22:32:02.546438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.546412    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lclj8\""
Sep 07 22:32:02.546680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.546662    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.546850 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.546834    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.547189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.547173    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.549452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.549434    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 22:32:02.549541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.549434    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 22:32:02.549541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.549495    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q92qn\""
Sep 07 22:32:02.550216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.550199    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.552216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.552197    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 22:32:02.552330 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.552312    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 22:32:02.552784 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.552764    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.552869 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.552826    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.552869 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.552837    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 22:32:02.553991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.553085    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb2sz\""
Sep 07 22:32:02.554114 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.554081    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 22:32:02.554255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.554238    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.555600 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555556    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-cnibin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555600 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555594    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-socket-dir-parent\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555618    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-kubelet\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555640    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-hostroot\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555664    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-conf-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd8d037-0fbc-43ed-995a-716b71f3edec-agent-certs\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555738    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-cni-binary-copy\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.555770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-k8s-cni-cncf-io\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555788    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-netns\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-multus-certs\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555836    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-etc-kubernetes\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555863    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/1a311daa-e34e-4591-a22a-5bf3e012a260-kube-api-access-chjrk\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8d037-0fbc-43ed-995a-716b71f3edec-konnectivity-ca\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555909    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db57f105-17e5-4bff-847b-e6e89b9b1eec-host-slash\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555941    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-system-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-multus\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.555990    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db57f105-17e5-4bff-847b-e6e89b9b1eec-iptables-alerter-script\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6vl\" (UniqueName: \"kubernetes.io/projected/db57f105-17e5-4bff-847b-e6e89b9b1eec-kube-api-access-6s6vl\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556051    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-bin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556079    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-daemon-config\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556101    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cb1275-a6c3-40bf-9c65-6a2128a40564-host\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cb1275-a6c3-40bf-9c65-6a2128a40564-serviceca\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.556246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556197    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8d8n\" (UniqueName: \"kubernetes.io/projected/75cb1275-a6c3-40bf-9c65-6a2128a40564-kube-api-access-k8d8n\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.556953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556688    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjslb\""
Sep 07 22:32:02.556953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.556935    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.557197 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.557106    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.559605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.559572    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.559684 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.559639    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:02.562766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.562745    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.565243 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.564982    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.565243 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.564995    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7zrdn\""
Sep 07 22:32:02.565243 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.565022    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 22:32:02.565433 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.565269    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 22:32:02.567618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.567242    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.567618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.567395    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.567763 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.567649    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 22:32:02.567763 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.567708    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-grvkj\""
Sep 07 22:32:02.567940 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.567765    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.569626 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.569604    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.569754 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.569669    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:02.570235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.570012    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 22:32:02.573413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.573333    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 22:32:02.575827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.573624    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:02.575827 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.573688    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:02.576302 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.576281    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r769h\""
Sep 07 22:32:02.582223 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.582121    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:27:01 +0000 UTC" deadline="2028-05-28 21:23:40.867618452 +0000 UTC"
Sep 07 22:32:02.582731 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.582682    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15094h51m38.285412947s"
Sep 07 22:32:02.644931 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.644903    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 22:32:02.657069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657045    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.657194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657085    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657119    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db57f105-17e5-4bff-847b-e6e89b9b1eec-iptables-alerter-script\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.657194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657168    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657188    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-tuned\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657210    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-ovn\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825bd\" (UniqueName: \"kubernetes.io/projected/6aa259ab-f410-44c8-a25c-e4aa9e05588c-kube-api-access-825bd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657253    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0676c7e-17b2-4596-b67e-9b5d54da77d4-tmp-dir\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657277    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-hostroot\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657309    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-conf-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657329    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-hostroot\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657335    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-sys\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657364    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-var-lib-kubelet\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657386    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-netd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657395    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-conf-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657410    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657459    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-registration-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657509    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/1a311daa-e34e-4591-a22a-5bf3e012a260-kube-api-access-chjrk\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657547    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2sgr\" (UniqueName: \"kubernetes.io/projected/6b032a2c-14fb-49f4-be42-528889c4d4fc-kube-api-access-k2sgr\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657575    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-config\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657623    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-device-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657646    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-kubelet-config\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657672    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657696    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-lib-modules\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657718    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-netns\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-systemd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657779    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-etc-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657837    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-log-socket\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.657929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-bin\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657914    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cb1275-a6c3-40bf-9c65-6a2128a40564-serviceca\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysconfig\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-tmp\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.657984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-sys-fs\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658008    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-multus-certs\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658032    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-system-cni-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658056    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-system-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658079    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-conf\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658082    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-multus-certs\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658100    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-run\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-systemd-units\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658159    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db57f105-17e5-4bff-847b-e6e89b9b1eec-iptables-alerter-script\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658172    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-node-log\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658198    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658222    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6vl\" (UniqueName: \"kubernetes.io/projected/db57f105-17e5-4bff-847b-e6e89b9b1eec-kube-api-access-6s6vl\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658247    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-daemon-config\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.658703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658250    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-system-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cb1275-a6c3-40bf-9c65-6a2128a40564-host\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658293    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0676c7e-17b2-4596-b67e-9b5d54da77d4-hosts-file\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658315    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8d8n\" (UniqueName: \"kubernetes.io/projected/75cb1275-a6c3-40bf-9c65-6a2128a40564-kube-api-access-k8d8n\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658328    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75cb1275-a6c3-40bf-9c65-6a2128a40564-serviceca\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658374    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-cnibin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75cb1275-a6c3-40bf-9c65-6a2128a40564-host\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658401    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-socket-dir-parent\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658427    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovn-node-metrics-cert\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658450    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrsv\" (UniqueName: \"kubernetes.io/projected/257a879d-f99b-4540-9185-342d65e29092-kube-api-access-nzrsv\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658478    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-cni-binary-copy\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658525    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-k8s-cni-cncf-io\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658533    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-cnibin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658554    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-netns\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658581    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-etc-kubernetes\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-kubelet\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-k8s-cni-cncf-io\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658630    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-script-lib\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.659569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658653    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-etc-kubernetes\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658655    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-host\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658659    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-socket-dir-parent\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658676    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-run-netns\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658679    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8g4\" (UniqueName: \"kubernetes.io/projected/251abc5e-221f-43c8-85cb-9107f713f1d6-kube-api-access-rm8g4\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-bin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658732    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-systemd\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658738    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-bin\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-binary-copy\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-dbus\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658796    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-kubelet\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658813    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd8d037-0fbc-43ed-995a-716b71f3edec-agent-certs\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658828    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8d037-0fbc-43ed-995a-716b71f3edec-konnectivity-ca\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658847    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658853    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-daemon-config\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658871    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-env-overrides\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.660458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a311daa-e34e-4591-a22a-5bf3e012a260-cni-binary-copy\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658919    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658941    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-kubelet\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658947    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.658991    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659022    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-kubernetes\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbnw\" (UniqueName: \"kubernetes.io/projected/99661a91-1abb-48c6-99f0-97ee822cab04-kube-api-access-wqbnw\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659061    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-slash\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659075    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-var-lib-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659075    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-multus-cni-dir\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659093    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-cnibin\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659111    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-socket-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659140    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbghz\" (UniqueName: \"kubernetes.io/projected/b0676c7e-17b2-4596-b67e-9b5d54da77d4-kube-api-access-mbghz\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659170    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db57f105-17e5-4bff-847b-e6e89b9b1eec-host-slash\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659206    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db57f105-17e5-4bff-847b-e6e89b9b1eec-host-slash\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-multus\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659245    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-modprobe-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.661286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659308    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a311daa-e34e-4591-a22a-5bf3e012a260-host-var-lib-cni-multus\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.662156 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8d037-0fbc-43ed-995a-716b71f3edec-konnectivity-ca\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.662156 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.659422    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 22:32:02.663510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.663488    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd8d037-0fbc-43ed-995a-716b71f3edec-agent-certs\") pod \"konnectivity-agent-9b7wn\" (UID: \"9cd8d037-0fbc-43ed-995a-716b71f3edec\") " pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.669758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.669734    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6vl\" (UniqueName: \"kubernetes.io/projected/db57f105-17e5-4bff-847b-e6e89b9b1eec-kube-api-access-6s6vl\") pod \"iptables-alerter-chkft\" (UID: \"db57f105-17e5-4bff-847b-e6e89b9b1eec\") " pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.670240 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.670218    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjrk\" (UniqueName: \"kubernetes.io/projected/1a311daa-e34e-4591-a22a-5bf3e012a260-kube-api-access-chjrk\") pod \"multus-9nfq6\" (UID: \"1a311daa-e34e-4591-a22a-5bf3e012a260\") " pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.671608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.671583    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8d8n\" (UniqueName: \"kubernetes.io/projected/75cb1275-a6c3-40bf-9c65-6a2128a40564-kube-api-access-k8d8n\") pod \"node-ca-mwfht\" (UID: \"75cb1275-a6c3-40bf-9c65-6a2128a40564\") " pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.759987 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.759947    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-systemd\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.759987 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.759983    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-binary-copy\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760006    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-dbus\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-env-overrides\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760064    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-systemd\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760078    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760108    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760152    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-kubernetes\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760201    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbnw\" (UniqueName: \"kubernetes.io/projected/99661a91-1abb-48c6-99f0-97ee822cab04-kube-api-access-wqbnw\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.760238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760199    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760242    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760265    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-etc-selinux\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-dbus\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-slash\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-slash\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-kubernetes\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760317    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-var-lib-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760348    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-cnibin\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-var-lib-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760371    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-socket-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbghz\" (UniqueName: \"kubernetes.io/projected/b0676c7e-17b2-4596-b67e-9b5d54da77d4-kube-api-access-mbghz\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760408    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-cnibin\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-modprobe-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760466    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.760704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760524    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-tuned\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760557    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-ovn\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760582    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-modprobe-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-ovn\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760598    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825bd\" (UniqueName: \"kubernetes.io/projected/6aa259ab-f410-44c8-a25c-e4aa9e05588c-kube-api-access-825bd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760597    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-env-overrides\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760615    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-binary-copy\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760523    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-socket-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760646    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.760678    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.760738    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:03.260718774 +0000 UTC m=+3.102987043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760763    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0676c7e-17b2-4596-b67e-9b5d54da77d4-tmp-dir\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760794    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-sys\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760810    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-d\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760860    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-var-lib-kubelet\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760886    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-netd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.761530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760914    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760930    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-sys\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760939    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-registration-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760930    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-var-lib-kubelet\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760964    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-netd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.760977    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2sgr\" (UniqueName: \"kubernetes.io/projected/6b032a2c-14fb-49f4-be42-528889c4d4fc-kube-api-access-k2sgr\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761001    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-registration-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-config\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-device-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761066    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0676c7e-17b2-4596-b67e-9b5d54da77d4-tmp-dir\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761105    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-kubelet-config\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761165    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-device-dir\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-lib-modules\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-netns\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.762374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761221    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-systemd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761245    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761268    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-etc-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.761286    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-log-socket\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761314    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-lib-modules\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761316    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-bin\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761221    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-kubelet-config\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.761335    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:03.261321793 +0000 UTC m=+3.103590067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761353    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-cni-bin\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761374    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-run-netns\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761380    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-run-systemd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761411    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/257a879d-f99b-4540-9185-342d65e29092-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761425    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysconfig\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761430    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-etc-openvswitch\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761459    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-log-socket\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761460    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-tmp\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.763391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761465    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysconfig\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-sys-fs\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761514    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-system-cni-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761537    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-conf\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761560    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-run\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761572    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-config\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-systemd-units\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761584    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-system-cni-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761577    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251abc5e-221f-43c8-85cb-9107f713f1d6-sys-fs\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761627    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-run\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761632    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-systemd-units\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761641    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-node-log\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761672    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-node-log\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761673    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-sysctl-conf\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761671    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761702    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0676c7e-17b2-4596-b67e-9b5d54da77d4-hosts-file\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761757    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0676c7e-17b2-4596-b67e-9b5d54da77d4-hosts-file\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.764038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovn-node-metrics-cert\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761790    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/257a879d-f99b-4540-9185-342d65e29092-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761802    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrsv\" (UniqueName: \"kubernetes.io/projected/257a879d-f99b-4540-9185-342d65e29092-kube-api-access-nzrsv\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761831    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-kubelet\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761875    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-script-lib\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-host\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761923    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa259ab-f410-44c8-a25c-e4aa9e05588c-host-kubelet\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.761942    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8g4\" (UniqueName: \"kubernetes.io/projected/251abc5e-221f-43c8-85cb-9107f713f1d6-kube-api-access-rm8g4\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.762064    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b032a2c-14fb-49f4-be42-528889c4d4fc-host\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.762391    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovnkube-script-lib\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.763635    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-etc-tuned\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.763693    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b032a2c-14fb-49f4-be42-528889c4d4fc-tmp\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.764588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.764025    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aa259ab-f410-44c8-a25c-e4aa9e05588c-ovn-node-metrics-cert\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.767446 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.767384    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:02.767446 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.767429    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:02.767446 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.767443    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:02.767651 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:02.767542    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:03.267523887 +0000 UTC m=+3.109792154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:02.769315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.769292    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbghz\" (UniqueName: \"kubernetes.io/projected/b0676c7e-17b2-4596-b67e-9b5d54da77d4-kube-api-access-mbghz\") pod \"node-resolver-jz776\" (UID: \"b0676c7e-17b2-4596-b67e-9b5d54da77d4\") " pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:02.769712 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.769688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-825bd\" (UniqueName: \"kubernetes.io/projected/6aa259ab-f410-44c8-a25c-e4aa9e05588c-kube-api-access-825bd\") pod \"ovnkube-node-kcfs8\" (UID: \"6aa259ab-f410-44c8-a25c-e4aa9e05588c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.770326 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.770306    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2sgr\" (UniqueName: \"kubernetes.io/projected/6b032a2c-14fb-49f4-be42-528889c4d4fc-kube-api-access-k2sgr\") pod \"tuned-5v5kv\" (UID: \"6b032a2c-14fb-49f4-be42-528889c4d4fc\") " pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.770584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.770564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8g4\" (UniqueName: \"kubernetes.io/projected/251abc5e-221f-43c8-85cb-9107f713f1d6-kube-api-access-rm8g4\") pod \"aws-ebs-csi-driver-node-gx9qj\" (UID: \"251abc5e-221f-43c8-85cb-9107f713f1d6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.770700 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.770567    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrsv\" (UniqueName: \"kubernetes.io/projected/257a879d-f99b-4540-9185-342d65e29092-kube-api-access-nzrsv\") pod \"multus-additional-cni-plugins-mcb92\" (UID: \"257a879d-f99b-4540-9185-342d65e29092\") " pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.770700 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.770567    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbnw\" (UniqueName: \"kubernetes.io/projected/99661a91-1abb-48c6-99f0-97ee822cab04-kube-api-access-wqbnw\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:02.793927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.793884    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"2f2c00fd5112014ad23844afadb2fd55","Type":"ContainerStarted","Data":"5366aaa215bfe977346b545c39edccee3226763943119251e13cfc2c47f9b6fc"}
Sep 07 22:32:02.795273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.795249    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"f669d2825f8c0e6d80ec2d9fbc641aea","Type":"ContainerStarted","Data":"edead60628e46ace9271bb2cc041e4ce596f32f7e9f956b6d3611d99fdb1bcc9"}
Sep 07 22:32:02.853667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.853638    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwfht"
Sep 07 22:32:02.865783 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.865762    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-chkft"
Sep 07 22:32:02.879316 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.879296    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nfq6"
Sep 07 22:32:02.886997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.886976    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:02.896438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.896419    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:02.903984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.903963    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v5kv"
Sep 07 22:32:02.911655 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.911637    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mcb92"
Sep 07 22:32:02.915794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.915775    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:02.919760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.919744    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj"
Sep 07 22:32:02.926399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:02.926382    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz776"
Sep 07 22:32:03.264861 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.264827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:03.265024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.264886    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:03.265024 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.264996    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:03.265100 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.265053    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:04.265039241 +0000 UTC m=+4.107307514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:03.265100 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.264997    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:03.265228 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.265159    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:04.265141528 +0000 UTC m=+4.107409785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:03.366116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.366086    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:03.366261 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.366219    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:03.366261 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.366233    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:03.366261 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.366241    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:03.366374 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.366291    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:04.366277386 +0000 UTC m=+4.208545630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:03.583349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.583215    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:27:01 +0000 UTC" deadline="2028-03-20 02:48:15.415258746 +0000 UTC"
Sep 07 22:32:03.583349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.583290    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13420h16m11.831973138s"
Sep 07 22:32:03.658603 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.658576    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd8d037_0fbc_43ed_995a_716b71f3edec.slice/crio-ff49c4586ca9fe3a429b85efbac03bc9627855f245070172dcd6f28e587e1d49 WatchSource:0}: Error finding container ff49c4586ca9fe3a429b85efbac03bc9627855f245070172dcd6f28e587e1d49: Status 404 returned error can't find the container with id ff49c4586ca9fe3a429b85efbac03bc9627855f245070172dcd6f28e587e1d49
Sep 07 22:32:03.660742 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.660577    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a311daa_e34e_4591_a22a_5bf3e012a260.slice/crio-23f4e66925189489c65bc5f00d29c612dab8dc540d06730f150c4e7ec9546ce0 WatchSource:0}: Error finding container 23f4e66925189489c65bc5f00d29c612dab8dc540d06730f150c4e7ec9546ce0: Status 404 returned error can't find the container with id 23f4e66925189489c65bc5f00d29c612dab8dc540d06730f150c4e7ec9546ce0
Sep 07 22:32:03.663962 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.663929    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cb1275_a6c3_40bf_9c65_6a2128a40564.slice/crio-e9b74c55e9184b9fdad794ed7195b9a98c9e1d4ac0a3bbb7223c35e55d09eb6c WatchSource:0}: Error finding container e9b74c55e9184b9fdad794ed7195b9a98c9e1d4ac0a3bbb7223c35e55d09eb6c: Status 404 returned error can't find the container with id e9b74c55e9184b9fdad794ed7195b9a98c9e1d4ac0a3bbb7223c35e55d09eb6c
Sep 07 22:32:03.664747 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.664727    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb57f105_17e5_4bff_847b_e6e89b9b1eec.slice/crio-2ad027adcd93a8ac6ddc9243906fdf121fc62c93c680521dc5ae0a541fe047b8 WatchSource:0}: Error finding container 2ad027adcd93a8ac6ddc9243906fdf121fc62c93c680521dc5ae0a541fe047b8: Status 404 returned error can't find the container with id 2ad027adcd93a8ac6ddc9243906fdf121fc62c93c680521dc5ae0a541fe047b8
Sep 07 22:32:03.666161 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.666111    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b032a2c_14fb_49f4_be42_528889c4d4fc.slice/crio-f4f4ceecf9a9968e3dcb04f24c3071f368adf0fb12d3252e9fd67edfc052c3c6 WatchSource:0}: Error finding container f4f4ceecf9a9968e3dcb04f24c3071f368adf0fb12d3252e9fd67edfc052c3c6: Status 404 returned error can't find the container with id f4f4ceecf9a9968e3dcb04f24c3071f368adf0fb12d3252e9fd67edfc052c3c6
Sep 07 22:32:03.667804 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.667783    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251abc5e_221f_43c8_85cb_9107f713f1d6.slice/crio-519ea109d4a63c5570cbf1753242ff592a5965041f0b3360669fbc49f1b4efb9 WatchSource:0}: Error finding container 519ea109d4a63c5570cbf1753242ff592a5965041f0b3360669fbc49f1b4efb9: Status 404 returned error can't find the container with id 519ea109d4a63c5570cbf1753242ff592a5965041f0b3360669fbc49f1b4efb9
Sep 07 22:32:03.669441 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.669193    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257a879d_f99b_4540_9185_342d65e29092.slice/crio-33947c29911a58a54cee95bbad1d1fcee9f556aff616678338f95815f0f6eacf WatchSource:0}: Error finding container 33947c29911a58a54cee95bbad1d1fcee9f556aff616678338f95815f0f6eacf: Status 404 returned error can't find the container with id 33947c29911a58a54cee95bbad1d1fcee9f556aff616678338f95815f0f6eacf
Sep 07 22:32:03.670047 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.670028    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa259ab_f410_44c8_a25c_e4aa9e05588c.slice/crio-670bb493c91e674b36b07ee921e21a0cd4426393b2fe36758d989de5419b50ff WatchSource:0}: Error finding container 670bb493c91e674b36b07ee921e21a0cd4426393b2fe36758d989de5419b50ff: Status 404 returned error can't find the container with id 670bb493c91e674b36b07ee921e21a0cd4426393b2fe36758d989de5419b50ff
Sep 07 22:32:03.670736 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:03.670543    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0676c7e_17b2_4596_b67e_9b5d54da77d4.slice/crio-eb5f575bffd67b617c08af9dd128a2c98dd54c3556d83555e8c9466a2537a137 WatchSource:0}: Error finding container eb5f575bffd67b617c08af9dd128a2c98dd54c3556d83555e8c9466a2537a137: Status 404 returned error can't find the container with id eb5f575bffd67b617c08af9dd128a2c98dd54c3556d83555e8c9466a2537a137
Sep 07 22:32:03.789924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.789902    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:03.790022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.789907    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:03.790076 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.790012    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:03.790154 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:03.790104    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:03.798646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.798625    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-chkft" event={"ID":"db57f105-17e5-4bff-847b-e6e89b9b1eec","Type":"ContainerStarted","Data":"2ad027adcd93a8ac6ddc9243906fdf121fc62c93c680521dc5ae0a541fe047b8"}
Sep 07 22:32:03.799712 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.799687    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9b7wn" event={"ID":"9cd8d037-0fbc-43ed-995a-716b71f3edec","Type":"ContainerStarted","Data":"ff49c4586ca9fe3a429b85efbac03bc9627855f245070172dcd6f28e587e1d49"}
Sep 07 22:32:03.800986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.800963    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj" event={"ID":"251abc5e-221f-43c8-85cb-9107f713f1d6","Type":"ContainerStarted","Data":"519ea109d4a63c5570cbf1753242ff592a5965041f0b3360669fbc49f1b4efb9"}
Sep 07 22:32:03.802190 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.802166    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"670bb493c91e674b36b07ee921e21a0cd4426393b2fe36758d989de5419b50ff"}
Sep 07 22:32:03.806168 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.806114    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwfht" event={"ID":"75cb1275-a6c3-40bf-9c65-6a2128a40564","Type":"ContainerStarted","Data":"e9b74c55e9184b9fdad794ed7195b9a98c9e1d4ac0a3bbb7223c35e55d09eb6c"}
Sep 07 22:32:03.807750 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.807726    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nfq6" event={"ID":"1a311daa-e34e-4591-a22a-5bf3e012a260","Type":"ContainerStarted","Data":"23f4e66925189489c65bc5f00d29c612dab8dc540d06730f150c4e7ec9546ce0"}
Sep 07 22:32:03.809642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.809617    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"2f2c00fd5112014ad23844afadb2fd55","Type":"ContainerStarted","Data":"8813c47c4d8a0225e76a87d04f5cbeb85dc911471090872f1e6f38e016b4abd5"}
Sep 07 22:32:03.810794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.810769    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz776" event={"ID":"b0676c7e-17b2-4596-b67e-9b5d54da77d4","Type":"ContainerStarted","Data":"eb5f575bffd67b617c08af9dd128a2c98dd54c3556d83555e8c9466a2537a137"}
Sep 07 22:32:03.811961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.811939    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerStarted","Data":"33947c29911a58a54cee95bbad1d1fcee9f556aff616678338f95815f0f6eacf"}
Sep 07 22:32:03.813300 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.813272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v5kv" event={"ID":"6b032a2c-14fb-49f4-be42-528889c4d4fc","Type":"ContainerStarted","Data":"f4f4ceecf9a9968e3dcb04f24c3071f368adf0fb12d3252e9fd67edfc052c3c6"}
Sep 07 22:32:03.821145 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:03.821087    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" podStartSLOduration=2.821076325 podStartE2EDuration="2.821076325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:03.821014546 +0000 UTC m=+3.663282812" watchObservedRunningTime="2026-09-07 22:32:03.821076325 +0000 UTC m=+3.663344591"
Sep 07 22:32:04.271032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.270930    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:04.271032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.270987    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:04.271282 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.271112    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:04.271282 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.271190    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:06.271171374 +0000 UTC m=+6.113439634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:04.271581 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.271559    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:04.271726 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.271613    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:06.271595519 +0000 UTC m=+6.113863764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:04.371813 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.371775    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:04.371978 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.371942    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:04.371978 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.371968    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:04.372076 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.371982    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:04.372076 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.372043    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:06.372024433 +0000 UTC m=+6.214292680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:04.793029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.792900    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:04.793781 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:04.793049    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:04.844023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.843435    2550 generic.go:356] "Generic (PLEG): container finished" podID="f669d2825f8c0e6d80ec2d9fbc641aea" containerID="d222b6d6512e681db6269923c43129fc0a38dad7c8b54fe41e2f0b970c78cd17" exitCode=0
Sep 07 22:32:04.844023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:04.843804    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"f669d2825f8c0e6d80ec2d9fbc641aea","Type":"ContainerDied","Data":"d222b6d6512e681db6269923c43129fc0a38dad7c8b54fe41e2f0b970c78cd17"}
Sep 07 22:32:05.789746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:05.789715    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:05.789962 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:05.789850    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:05.789962 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:05.789926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:05.790096 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:05.790008    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:05.883424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:05.883322    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"f669d2825f8c0e6d80ec2d9fbc641aea","Type":"ContainerStarted","Data":"1b40e7fe5f54584d2b7996108a0a61a87e1385e44e7b60f9f1bd9ee6e94f49d5"}
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:06.288056    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:06.288140    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.288270    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.288329    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:10.288309134 +0000 UTC m=+10.130577395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.288698    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:06.288821 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.288743    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:10.288728693 +0000 UTC m=+10.130996952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:06.389232 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:06.389195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:06.389416 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.389401    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:06.389479 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.389423    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:06.389479 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.389437    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:06.389588 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.389512    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:10.389491637 +0000 UTC m=+10.231759905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:06.789888 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:06.789863    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:06.790043 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:06.790001    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:07.789747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:07.789683    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:07.790253 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:07.789805    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:07.790253 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:07.790153    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:07.790253 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:07.790245    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:08.790822 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:08.790792    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:08.791271 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:08.790935    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:09.789753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:09.789716    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:09.789919 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:09.789850    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:09.789919 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:09.789898    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:09.790039 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:09.790019    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:10.320551 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:10.320508    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:10.321021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:10.320616    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:10.321021 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.320708    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:10.321021 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.320744    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:10.321021 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.320792    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:18.320770153 +0000 UTC m=+18.163038420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:10.321021 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.320813    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:18.320803039 +0000 UTC m=+18.163071289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:10.421961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:10.421921    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:10.422230 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.422164    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:10.422230 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.422196    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:10.422230 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.422210    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:10.422480 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.422312    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:18.422291078 +0000 UTC m=+18.264559328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:10.791888 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:10.791416    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:10.791888 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:10.791547    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:11.789943 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:11.789904    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:11.790478 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:11.790036    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:11.790478 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:11.790062    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:11.790478 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:11.790190    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:12.789708 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:12.789670    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:12.789895 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:12.789788    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:13.789971 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:13.789941    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:13.790452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:13.789941    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:13.790452 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:13.790056    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:13.790452 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:13.790102    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:14.789746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:14.789670    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:14.789899 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:14.789793    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:15.790331 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:15.790296    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:15.790762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:15.790339    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:15.790762 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:15.790435    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:15.790762 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:15.790575    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:16.790459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:16.790428    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:16.790859 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:16.790535    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:17.789621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:17.789583    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:17.789853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:17.789586    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:17.789853 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:17.789718    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:17.789853 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:17.789782    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:18.377343 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:18.377310    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:18.377803 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.377454    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:18.377803 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.377519    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:34.377499811 +0000 UTC m=+34.219768078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:18.377803 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:18.377513    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:18.377803 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.377641    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:18.377803 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.377700    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:34.377683347 +0000 UTC m=+34.219951599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:18.477982 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:18.477925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:18.478153 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.478069    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:18.478153 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.478100    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:18.478153 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.478114    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:18.478276 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.478199    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:34.47817913 +0000 UTC m=+34.320447391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:18.789757 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:18.789656    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:18.789916 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:18.789799    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:19.789438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:19.789405    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:19.789879 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:19.789539    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:19.789879 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:19.789598    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:19.789879 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:19.789703    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:20.790941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:20.790912    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:20.791369 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:20.791020    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:21.789946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.789764    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:21.790091 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:21.790065    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:21.790233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.790212    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:21.790394 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:21.790368    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:21.919368 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.919328    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="c528a0d6bc2bcf61bd76af1adeaf01a7c30ac7d75c862e47ebfe8d6353769715" exitCode=0
Sep 07 22:32:21.919912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.919436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"c528a0d6bc2bcf61bd76af1adeaf01a7c30ac7d75c862e47ebfe8d6353769715"}
Sep 07 22:32:21.921630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.921598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v5kv" event={"ID":"6b032a2c-14fb-49f4-be42-528889c4d4fc","Type":"ContainerStarted","Data":"5235fd0462fef6956e5f8f231c17572e4eecf973e3c1070ff089fccd0508557d"}
Sep 07 22:32:21.923360 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.923338    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9b7wn" event={"ID":"9cd8d037-0fbc-43ed-995a-716b71f3edec","Type":"ContainerStarted","Data":"870d380fc658fbaafdd74fe5a3f53ed6bc61b782a5c4cf21c177fbda60392b6a"}
Sep 07 22:32:21.928057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.927999    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj" event={"ID":"251abc5e-221f-43c8-85cb-9107f713f1d6","Type":"ContainerStarted","Data":"19199703a100770d6086bd951ea497875ff6ed3472aaf639b4edb2381043c6e8"}
Sep 07 22:32:21.930304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.930282    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"102bc92f4f7dce5bd9683e63dd1df7396f33d095b4f4e34a46fbd150afb0fbbc"}
Sep 07 22:32:21.930495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.930344    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"49a08ea3e5c4ab67920d4f3411c09a5bf78349c715ae5bf50bad9a514d325ba7"}
Sep 07 22:32:21.930495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.930384    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"b2b1647ecb5b4aa75f808113811c76077769b2e98103caac986de646a7e7e81f"}
Sep 07 22:32:21.931999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.931970    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwfht" event={"ID":"75cb1275-a6c3-40bf-9c65-6a2128a40564","Type":"ContainerStarted","Data":"387bab3e9b62a2c560e5e64e9345e229c6099b680a47de159c435d62c6a6a76e"}
Sep 07 22:32:21.933515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.933487    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nfq6" event={"ID":"1a311daa-e34e-4591-a22a-5bf3e012a260","Type":"ContainerStarted","Data":"74350f57d3742ef667a6fdd32536339dd8f5472caf3ceac5de8d03d29dff3869"}
Sep 07 22:32:21.935066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.935037    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz776" event={"ID":"b0676c7e-17b2-4596-b67e-9b5d54da77d4","Type":"ContainerStarted","Data":"1ab66736c45c5cd2dcfd2fe28ebdbb19ee53ebea28b235e8cdb5c6a91692c234"}
Sep 07 22:32:21.939504 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.939465    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" podStartSLOduration=20.939454242 podStartE2EDuration="20.939454242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:05.897718429 +0000 UTC m=+5.739986697" watchObservedRunningTime="2026-09-07 22:32:21.939454242 +0000 UTC m=+21.781722509"
Sep 07 22:32:21.952035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.952001    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mwfht" podStartSLOduration=4.075596304 podStartE2EDuration="21.951993557s" totalImagesPullingTime="17.876397253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:21.951845285 +0000 UTC m=+21.794113552" watchObservedRunningTime="2026-09-07 22:32:21.951993557 +0000 UTC m=+21.794261824"
Sep 07 22:32:21.966922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.966888    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9nfq6" podStartSLOduration=4.027745222 podStartE2EDuration="21.966875801s" totalImagesPullingTime="17.939130579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:21.966471558 +0000 UTC m=+21.808739837" watchObservedRunningTime="2026-09-07 22:32:21.966875801 +0000 UTC m=+21.809144069"
Sep 07 22:32:21.980629 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.980594    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9b7wn" podStartSLOduration=11.52203133 podStartE2EDuration="21.980582931s" totalImagesPullingTime="10.458551601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:21.980531992 +0000 UTC m=+21.822800260" watchObservedRunningTime="2026-09-07 22:32:21.980582931 +0000 UTC m=+21.822851198"
Sep 07 22:32:21.996856 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:21.996823    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5v5kv" podStartSLOduration=4.120117616 podStartE2EDuration="21.996812157s" totalImagesPullingTime="17.876694541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:21.996728711 +0000 UTC m=+21.838996990" watchObservedRunningTime="2026-09-07 22:32:21.996812157 +0000 UTC m=+21.839080424"
Sep 07 22:32:22.008730 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.008698    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jz776" podStartSLOduration=3.158737157 podStartE2EDuration="21.008687577s" totalImagesPullingTime="17.84995042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:22.008678927 +0000 UTC m=+21.850947194" watchObservedRunningTime="2026-09-07 22:32:22.008687577 +0000 UTC m=+21.850955844"
Sep 07 22:32:22.789884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.789848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:22.790078 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:22.789968    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:22.941486 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.941449    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"b3e7b24ea5249045234cb750fff4de192f6fd079ddf67225ea5743a40a068fd6"}
Sep 07 22:32:22.941981 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.941549    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"6844a423a80940cc66ce4518adad70c145b93f5719dbe73be02d1f660034425b"}
Sep 07 22:32:22.941981 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.941605    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"0ea276ebde6534140ce95ce0de3b94a8850d3bbd9452d5d96ea50305be47eee5"}
Sep 07 22:32:22.943574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.943548    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-chkft" event={"ID":"db57f105-17e5-4bff-847b-e6e89b9b1eec","Type":"ContainerStarted","Data":"72eea6126f878499c0924228631a58387ca3b8a05b0074b3c69a44814ef3d532"}
Sep 07 22:32:22.956805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.956766    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-chkft" podStartSLOduration=5.082239252 podStartE2EDuration="22.956751808s" totalImagesPullingTime="17.874512556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:22.956495887 +0000 UTC m=+22.798764165" watchObservedRunningTime="2026-09-07 22:32:22.956751808 +0000 UTC m=+22.799020074"
Sep 07 22:32:22.990913 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:22.990883    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 22:32:23.758236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.758099    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T22:32:22.99090963Z","UUID":"89c95c1d-c863-4ac3-a8c7-4aa6eda5b28b","Handler":null,"Name":"","Endpoint":""}
Sep 07 22:32:23.759792 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.759770    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 22:32:23.759950 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.759807    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 22:32:23.790568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.790474    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:23.790724 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:23.790569    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:23.790802 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.790787    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:23.790887 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:23.790854    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:23.948763 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:23.948718    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj" event={"ID":"251abc5e-221f-43c8-85cb-9107f713f1d6","Type":"ContainerStarted","Data":"8a98fc2eea36804fb5ccf6ad7b11e1557c61c141af0858aba58186d53b24d91a"}
Sep 07 22:32:24.684752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.684675    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:24.685273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.685248    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:24.790444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.790414    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:24.790594 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:24.790529    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:24.953396 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.953313    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj" event={"ID":"251abc5e-221f-43c8-85cb-9107f713f1d6","Type":"ContainerStarted","Data":"e317c6a3b1a7eb957e11d4ce5c0744d2a65018bd252e36cbd8666c05509b2fb9"}
Sep 07 22:32:24.957190 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.957160    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"b252ee5fba4b083f6861ca9da273d45d0c127fdf8b28e9dcf6093d36104f85b3"}
Sep 07 22:32:24.957378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.957362    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:24.957911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.957893    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9b7wn"
Sep 07 22:32:24.970560 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:24.970516    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gx9qj" podStartSLOduration=3.844932317 podStartE2EDuration="23.970502069s" totalImagesPullingTime="20.125569752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:24.970455968 +0000 UTC m=+24.812724234" watchObservedRunningTime="2026-09-07 22:32:24.970502069 +0000 UTC m=+24.812770335"
Sep 07 22:32:25.790107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:25.790073    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:25.790107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:25.790096    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:25.790338 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:25.790212    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:25.790338 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:25.790327    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:26.789754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.789587    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:26.790052 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:26.789848    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:26.964966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.964887    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" event={"ID":"6aa259ab-f410-44c8-a25c-e4aa9e05588c","Type":"ContainerStarted","Data":"60b6454af853dd31ee69580c6242210aaea4bb71ceff33cdc0b9f099d09659a3"}
Sep 07 22:32:26.965119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.965102    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:26.966872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.966849    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="cc7e6f4dfcebc94d16304b567cf5148965ac3ecf364dd79d1f9780ffd1864805" exitCode=0
Sep 07 22:32:26.966976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.966932    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"cc7e6f4dfcebc94d16304b567cf5148965ac3ecf364dd79d1f9780ffd1864805"}
Sep 07 22:32:26.980305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.980288    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:26.988613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:26.988573    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8" podStartSLOduration=9.084823461 podStartE2EDuration="26.988559776s" totalImagesPullingTime="17.903736315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:26.986975107 +0000 UTC m=+26.829243373" watchObservedRunningTime="2026-09-07 22:32:26.988559776 +0000 UTC m=+26.830828042"
Sep 07 22:32:27.791156 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.791030    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:27.791611 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:27.791184    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:27.791611 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.791582    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:27.791690 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:27.791675    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:27.971298 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.971264    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerStarted","Data":"5a9d1f845bb81db64c96476725dfc56cd879c49558dcefc3cceab8e10ef0191c"}
Sep 07 22:32:27.972122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.971981    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:27.972122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.972068    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:27.985505 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:27.985482    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:32:28.424923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.424893    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7772"]
Sep 07 22:32:28.425033 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.424997    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:28.425100 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:28.425077    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:28.428439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.428415    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2k4w"]
Sep 07 22:32:28.428545 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.428515    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:28.428628 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:28.428609    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:28.429055 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.429033    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hds5x"]
Sep 07 22:32:28.429188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.429147    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:28.429265 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:28.429248    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:28.975613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.975581    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="5a9d1f845bb81db64c96476725dfc56cd879c49558dcefc3cceab8e10ef0191c" exitCode=0
Sep 07 22:32:28.976410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:28.975670    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"5a9d1f845bb81db64c96476725dfc56cd879c49558dcefc3cceab8e10ef0191c"}
Sep 07 22:32:29.790434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:29.790354    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:29.790549 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:29.790445    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:29.790549 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:29.790501    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:29.790627 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:29.790589    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:29.979759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:29.979731    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="4abf6a7c6c6841f561e6a47648a93642fbecdf73961ae99a98219942c6ee7cd0" exitCode=0
Sep 07 22:32:29.980077 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:29.979806    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"4abf6a7c6c6841f561e6a47648a93642fbecdf73961ae99a98219942c6ee7cd0"}
Sep 07 22:32:30.790842 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:30.790762    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:30.791024 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:30.790867    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:31.404723 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:31.404649    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:31.488247 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:31.488219    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:31.789725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:31.789681    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:31.789725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:31.789710    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:31.789938 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:31.789807    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:31.790000 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:31.789978    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:32.790487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:32.790446    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:32.790893 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:32.790586    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7772" podUID="c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2"
Sep 07 22:32:33.789551 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:33.789517    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:33.789770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:33.789525    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:33.789770 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:33.789639    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g2k4w" podUID="4d2b780d-b803-45c9-b9b7-a6092b85ad76"
Sep 07 22:32:33.789770 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:33.789737    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hds5x" podUID="99661a91-1abb-48c6-99f0-97ee822cab04"
Sep 07 22:32:34.396094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.396063    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:34.396456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.396120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:34.396456 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.396216    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:34.396456 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.396251    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:34.396456 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.396288    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:06.396268717 +0000 UTC m=+66.238536970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:34.396456 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.396302    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret podName:c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:06.396296429 +0000 UTC m=+66.238564674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret") pod "global-pull-secret-syncer-c7772" (UID: "c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:34.397227 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.397209    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeReady"
Sep 07 22:32:34.397343 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.397332    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 22:32:34.436810 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.436784    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-svkjt"]
Sep 07 22:32:34.451984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.451953    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k457m"]
Sep 07 22:32:34.452104 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.452085    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.454740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.454714    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 22:32:34.454953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.454930    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 22:32:34.455027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.454996    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:32:34.474295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.474273    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-svkjt"]
Sep 07 22:32:34.474400 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.474315    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k457m"]
Sep 07 22:32:34.474460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.474410    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.477053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.477029    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zz6fl\""
Sep 07 22:32:34.477194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.477029    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 22:32:34.477194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.477030    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 22:32:34.477194 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.477061    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 22:32:34.497353 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.497329    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:34.497442 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.497431    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:34.497484 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.497446    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:34.497484 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.497456    2550 projected.go:196] Error preparing data for projected volume kube-api-access-plfnl for pod openshift-network-diagnostics/network-check-target-g2k4w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:34.497546 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.497508    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl podName:4d2b780d-b803-45c9-b9b7-a6092b85ad76 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:06.497495603 +0000 UTC m=+66.339763861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-plfnl" (UniqueName: "kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl") pod "network-check-target-g2k4w" (UID: "4d2b780d-b803-45c9-b9b7-a6092b85ad76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:34.550101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.550074    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:34.598210 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-config-volume\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.598295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598213    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2wj\" (UniqueName: \"kubernetes.io/projected/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-kube-api-access-qm2wj\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.598295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598243    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnj9\" (UniqueName: \"kubernetes.io/projected/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-kube-api-access-qhnj9\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.598295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598260    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-tmp-dir\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.598295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598286    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.598424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.598340    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-cert\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.599978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.599959    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:34.699286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699256    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2wj\" (UniqueName: \"kubernetes.io/projected/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-kube-api-access-qm2wj\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.699419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699293    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnj9\" (UniqueName: \"kubernetes.io/projected/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-kube-api-access-qhnj9\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.699419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-tmp-dir\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.699419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699332    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.699419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699361    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-cert\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.699419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699389    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-config-volume\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.699654 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.699521    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:34.699654 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:34.699591    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:35.199570774 +0000 UTC m=+35.041839027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:34.699918 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.699901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-tmp-dir\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.700157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.700115    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-config-volume\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.703608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.703587    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-cert\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.707343 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.707314    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnj9\" (UniqueName: \"kubernetes.io/projected/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-kube-api-access-qhnj9\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:34.707428 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.707358    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2wj\" (UniqueName: \"kubernetes.io/projected/6ec8a08b-16e8-4f9f-89a0-75331e4edb80-kube-api-access-qm2wj\") pod \"ingress-canary-k457m\" (UID: \"6ec8a08b-16e8-4f9f-89a0-75331e4edb80\") " pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.782572 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.782547    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k457m"
Sep 07 22:32:34.789824 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.789803    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:32:34.792347 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.792330    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:32:34.994029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.994003    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="71a4764426325e49d084ce553f3ee89cd218f10daceeb407e12871c6eca5f196" exitCode=0
Sep 07 22:32:34.994189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:34.994059    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"71a4764426325e49d084ce553f3ee89cd218f10daceeb407e12871c6eca5f196"}
Sep 07 22:32:35.090123 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.090098    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k457m"]
Sep 07 22:32:35.092944 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:32:35.092922    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec8a08b_16e8_4f9f_89a0_75331e4edb80.slice/crio-55b397e03f9fdcfba1b323ff264ce0cad379d0ba627f7aa56f864b56aa160871 WatchSource:0}: Error finding container 55b397e03f9fdcfba1b323ff264ce0cad379d0ba627f7aa56f864b56aa160871: Status 404 returned error can't find the container with id 55b397e03f9fdcfba1b323ff264ce0cad379d0ba627f7aa56f864b56aa160871
Sep 07 22:32:35.203288 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.203259    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:35.203415 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:35.203396    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:35.203463 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:35.203454    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:36.203438399 +0000 UTC m=+36.045706644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:35.339637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.339605    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:35.440506 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.440482    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:35.789820 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.789710    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:32:35.790044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.789944    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:32:35.792386 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.792362    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:32:35.792657 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.792637    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:32:35.792927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.792909    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:32:35.793006 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.792987    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:32:35.793075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.793036    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:32:35.908836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.908802    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:35.958531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.958502    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:35.997634 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:35.997598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k457m" event={"ID":"6ec8a08b-16e8-4f9f-89a0-75331e4edb80","Type":"ContainerStarted","Data":"55b397e03f9fdcfba1b323ff264ce0cad379d0ba627f7aa56f864b56aa160871"}
Sep 07 22:32:36.001458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:36.001428    2550 generic.go:356] "Generic (PLEG): container finished" podID="257a879d-f99b-4540-9185-342d65e29092" containerID="c556a183007a3d660c5bced8d423609e2073511cf202e7e433647c6e77a36feb" exitCode=0
Sep 07 22:32:36.001553 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:36.001465    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerDied","Data":"c556a183007a3d660c5bced8d423609e2073511cf202e7e433647c6e77a36feb"}
Sep 07 22:32:36.209112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:36.209081    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:36.209246 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:36.209225    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:36.209308 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:36.209290    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:38.209269952 +0000 UTC m=+38.051538201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:36.656808 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:36.656760    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:36.707901 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:36.707860    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:37.007730 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.007698    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mcb92" event={"ID":"257a879d-f99b-4540-9185-342d65e29092","Type":"ContainerStarted","Data":"d838fa1e6b19146cae9086ce513e2e323c3b0c59a7181d8b7c5930d7c98ee50f"}
Sep 07 22:32:37.028421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.028373    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mcb92" podStartSLOduration=7.826403442 podStartE2EDuration="36.028357047s" totalImagesPullingTime="28.201953605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:37.0267061 +0000 UTC m=+36.868974390" watchObservedRunningTime="2026-09-07 22:32:37.028357047 +0000 UTC m=+36.870625314"
Sep 07 22:32:37.145960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.145933    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:37.193582 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.193546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:37.503631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.503608    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:37.877178 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.877069    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:37.977603 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:37.977577    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:38.014512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:38.014480    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k457m" event={"ID":"6ec8a08b-16e8-4f9f-89a0-75331e4edb80","Type":"ContainerStarted","Data":"79f5d21fd3517593ff8cde79df093277aed84932ce893cf3076815daa7f0ab8d"}
Sep 07 22:32:38.030651 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:38.030612    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k457m" podStartSLOduration=1.661976214 podStartE2EDuration="4.030598758s" totalImagesPullingTime="2.368622544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:38.03003472 +0000 UTC m=+37.872302989" watchObservedRunningTime="2026-09-07 22:32:38.030598758 +0000 UTC m=+37.872867077"
Sep 07 22:32:38.221651 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:38.221620    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:38.221774 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:38.221717    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:38.221774 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:38.221768    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:42.221754894 +0000 UTC m=+42.064023143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:38.284774 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:38.284741    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:39.723541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:39.723511    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:39.782502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:39.782475    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:42.248254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:42.248214    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:42.248717 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:42.248391    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:42.248717 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:42.248470    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:50.248448205 +0000 UTC m=+50.090716474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:46.778079 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:46.778001    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:46.825414 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:46.825389    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:47.992162 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:47.992115    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:48.050890 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:48.050858    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:50.302377 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:50.302347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:32:50.302826 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:50.302465    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:50.302826 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:32:50.302519    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:06.30250465 +0000 UTC m=+66.144772894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:51.421256 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:51.421220    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:51.526017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:51.525971    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:52.761057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:52.761025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:52.809785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:52.809745    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:58.894247 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:58.894202    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:58.943000 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:58.942973    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:32:59.990636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:32:59.990609    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kcfs8"
Sep 07 22:33:00.136298 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:00.136270    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:00.192985 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:00.192961    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:06.309811 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.309765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:06.310205 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:06.309909    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:33:06.310205 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:06.309984    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls podName:b74b79f4-9aac-4c4d-998c-cd09b9e02b53 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:38.309963523 +0000 UTC m=+98.152231785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls") pod "dns-default-svkjt" (UID: "b74b79f4-9aac-4c4d-998c-cd09b9e02b53") : secret "dns-default-metrics-tls" not found
Sep 07 22:33:06.410678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.410640    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:33:06.410812 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.410686    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:33:06.413683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.413665    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:33:06.413734 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.413677    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:33:06.421795 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:06.421772    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 22:33:06.421904 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:06.421845    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs podName:99661a91-1abb-48c6-99f0-97ee822cab04 nodeName:}" failed. No retries permitted until 2026-09-07 22:34:10.421823574 +0000 UTC m=+130.264091821 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs") pod "network-metrics-daemon-hds5x" (UID: "99661a91-1abb-48c6-99f0-97ee822cab04") : secret "metrics-daemon-secret" not found
Sep 07 22:33:06.424866 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.424849    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2-original-pull-secret\") pod \"global-pull-secret-syncer-c7772\" (UID: \"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2\") " pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:33:06.511936 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.511906    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:33:06.514321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.514304    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:33:06.524207 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.524188    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:33:06.535094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.535071    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfnl\" (UniqueName: \"kubernetes.io/projected/4d2b780d-b803-45c9-b9b7-a6092b85ad76-kube-api-access-plfnl\") pod \"network-check-target-g2k4w\" (UID: \"4d2b780d-b803-45c9-b9b7-a6092b85ad76\") " pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:33:06.598052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.597994    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7772"
Sep 07 22:33:06.710946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.710920    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:33:06.719022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.719004    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:33:06.861358 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.861289    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7772"]
Sep 07 22:33:06.864660 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:06.864633    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5035533_3c1e_4f5a_a5f2_4d07e4ee8eb2.slice/crio-9f29b72df7de9fba91203be9c671c0727a78993619d39b17856e2987e0926b20 WatchSource:0}: Error finding container 9f29b72df7de9fba91203be9c671c0727a78993619d39b17856e2987e0926b20: Status 404 returned error can't find the container with id 9f29b72df7de9fba91203be9c671c0727a78993619d39b17856e2987e0926b20
Sep 07 22:33:06.979412 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:06.979380    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g2k4w"]
Sep 07 22:33:06.981768 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:06.981743    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2b780d_b803_45c9_b9b7_a6092b85ad76.slice/crio-5b6f65ae41f55708cfe2cfb8a46339456ae6cac3ab9e404a0264831d32cb20af WatchSource:0}: Error finding container 5b6f65ae41f55708cfe2cfb8a46339456ae6cac3ab9e404a0264831d32cb20af: Status 404 returned error can't find the container with id 5b6f65ae41f55708cfe2cfb8a46339456ae6cac3ab9e404a0264831d32cb20af
Sep 07 22:33:07.082732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.082702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7772" event={"ID":"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2","Type":"ContainerStarted","Data":"9f29b72df7de9fba91203be9c671c0727a78993619d39b17856e2987e0926b20"}
Sep 07 22:33:07.083788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.083769    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2k4w" event={"ID":"4d2b780d-b803-45c9-b9b7-a6092b85ad76","Type":"ContainerStarted","Data":"5b6f65ae41f55708cfe2cfb8a46339456ae6cac3ab9e404a0264831d32cb20af"}
Sep 07 22:33:07.894720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.894677    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"]
Sep 07 22:33:07.896638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.896613    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"
Sep 07 22:33:07.899724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.899158    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qxjhl\""
Sep 07 22:33:07.899724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.899239    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:07.900355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.900249    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.907601 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.907577    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"]
Sep 07 22:33:07.921627 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.921565    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/ab1e0fd0-df4c-4232-80ff-b3c222b86ac6-kube-api-access-njv5j\") pod \"volume-data-source-validator-695c895bd6-xltb9\" (UID: \"ab1e0fd0-df4c-4232-80ff-b3c222b86ac6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"
Sep 07 22:33:07.993159 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.993114    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5789d6f94b-r48qj"]
Sep 07 22:33:07.994986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.994959    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p46w7"]
Sep 07 22:33:07.995116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.995092    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:07.996712 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.996682    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7"
Sep 07 22:33:07.997620 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.997601    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 07 22:33:07.997994 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.997965    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 22:33:07.998077 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.998019    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 07 22:33:07.998367 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.998349    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 07 22:33:07.998469 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.998402    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 07 22:33:07.998528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.998356    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2qfwj\""
Sep 07 22:33:07.998670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.998650    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.999369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:07.999347    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gpnhj\""
Sep 07 22:33:08.009312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.008779    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p46w7"]
Sep 07 22:33:08.009821 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.009801    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5789d6f94b-r48qj"]
Sep 07 22:33:08.022454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022429    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdt4\" (UniqueName: \"kubernetes.io/projected/438316bc-84ec-4b46-827d-ce96b271d110-kube-api-access-vtdt4\") pod \"network-check-source-685456655f-p46w7\" (UID: \"438316bc-84ec-4b46-827d-ce96b271d110\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7"
Sep 07 22:33:08.022563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022472    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppwv\" (UniqueName: \"kubernetes.io/projected/2094faa6-7b9d-4f4d-b65a-105f8a62e214-kube-api-access-4ppwv\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.022563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022506    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.022563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022539    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-default-certificate\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.022735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022586    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.022735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022649    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-stats-auth\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.022811 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.022735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/ab1e0fd0-df4c-4232-80ff-b3c222b86ac6-kube-api-access-njv5j\") pod \"volume-data-source-validator-695c895bd6-xltb9\" (UID: \"ab1e0fd0-df4c-4232-80ff-b3c222b86ac6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"
Sep 07 22:33:08.034920 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.034892    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/ab1e0fd0-df4c-4232-80ff-b3c222b86ac6-kube-api-access-njv5j\") pod \"volume-data-source-validator-695c895bd6-xltb9\" (UID: \"ab1e0fd0-df4c-4232-80ff-b3c222b86ac6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"
Sep 07 22:33:08.089763 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.089734    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7qqt"]
Sep 07 22:33:08.091838 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.091813    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"]
Sep 07 22:33:08.091969 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.091901    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.093983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.093708    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.095288 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.095244    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-znmd2\""
Sep 07 22:33:08.095740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.095495    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 07 22:33:08.095740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.095546    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 07 22:33:08.096613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.096414    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-blz88\""
Sep 07 22:33:08.096613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.096468    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 07 22:33:08.096613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.096482    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:08.096613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.096525    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 07 22:33:08.096952 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.096738    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 07 22:33:08.108074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.107973    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"]
Sep 07 22:33:08.108387 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.108353    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7qqt"]
Sep 07 22:33:08.123458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123431    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.123580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123479    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdt4\" (UniqueName: \"kubernetes.io/projected/438316bc-84ec-4b46-827d-ce96b271d110-kube-api-access-vtdt4\") pod \"network-check-source-685456655f-p46w7\" (UID: \"438316bc-84ec-4b46-827d-ce96b271d110\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7"
Sep 07 22:33:08.123580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4kw\" (UniqueName: \"kubernetes.io/projected/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-kube-api-access-wg4kw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.123580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123551    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppwv\" (UniqueName: \"kubernetes.io/projected/2094faa6-7b9d-4f4d-b65a-105f8a62e214-kube-api-access-4ppwv\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123584    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123609    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-default-certificate\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123671    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af3b600-3fac-4e9f-b577-24d9db07bf35-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.123752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-stats-auth\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.124095 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.123814    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.124095 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.124019    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.623998198 +0000 UTC m=+68.466266464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:08.124394 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.124344    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 22:33:08.124635 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.124615    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.624554227 +0000 UTC m=+68.466822487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : secret "router-metrics-certs-default" not found
Sep 07 22:33:08.126850 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.126825    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-default-certificate\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.127322 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.127295    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-stats-auth\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.137574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.137548    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:08.137948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.137890    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdt4\" (UniqueName: \"kubernetes.io/projected/438316bc-84ec-4b46-827d-ce96b271d110-kube-api-access-vtdt4\") pod \"network-check-source-685456655f-p46w7\" (UID: \"438316bc-84ec-4b46-827d-ce96b271d110\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7"
Sep 07 22:33:08.138192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.138114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppwv\" (UniqueName: \"kubernetes.io/projected/2094faa6-7b9d-4f4d-b65a-105f8a62e214-kube-api-access-4ppwv\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.208212 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.208182    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"
Sep 07 22:33:08.218845 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.218783    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:08.224747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.224722    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.224856 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.224765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.224856 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.224785    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4kw\" (UniqueName: \"kubernetes.io/projected/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-kube-api-access-wg4kw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.224968 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.224939    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:08.225060 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.225035    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.725005205 +0000 UTC m=+68.567273451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:08.225314 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.225287    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.225391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.225342    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af3b600-3fac-4e9f-b577-24d9db07bf35-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.225525 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.225290    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.226058 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.226015    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af3b600-3fac-4e9f-b577-24d9db07bf35-nginx-conf\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.227687 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.227663    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.236929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.236906    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4kw\" (UniqueName: \"kubernetes.io/projected/fdac8917-90cc-4c89-8e2e-3dccfc77a3bc-kube-api-access-wg4kw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t92v8\" (UID: \"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.315560 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.315522    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7"
Sep 07 22:33:08.318827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.318751    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:08.366754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.366723    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:08.412307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.412220    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"
Sep 07 22:33:08.495343 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.495267    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9"]
Sep 07 22:33:08.498741 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:08.498702    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1e0fd0_df4c_4232_80ff_b3c222b86ac6.slice/crio-b5104034047251133af9da12dff1819510026e3f55d816598fdfaa81b79a62bd WatchSource:0}: Error finding container b5104034047251133af9da12dff1819510026e3f55d816598fdfaa81b79a62bd: Status 404 returned error can't find the container with id b5104034047251133af9da12dff1819510026e3f55d816598fdfaa81b79a62bd
Sep 07 22:33:08.590029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.589959    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p46w7"]
Sep 07 22:33:08.592780 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:08.592753    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438316bc_84ec_4b46_827d_ce96b271d110.slice/crio-535f62fe8c8d7340daad9613b92e21e2df990a07bccd639a2e7fec1aa2da1b8d WatchSource:0}: Error finding container 535f62fe8c8d7340daad9613b92e21e2df990a07bccd639a2e7fec1aa2da1b8d: Status 404 returned error can't find the container with id 535f62fe8c8d7340daad9613b92e21e2df990a07bccd639a2e7fec1aa2da1b8d
Sep 07 22:33:08.628825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.628792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.628928 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.628899    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:08.628994 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.628951    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.628928192 +0000 UTC m=+69.471196453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:08.629058 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.629015    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 22:33:08.629106 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.629067    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.629051444 +0000 UTC m=+69.471319698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : secret "router-metrics-certs-default" not found
Sep 07 22:33:08.689802 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.689731    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8"]
Sep 07 22:33:08.693144 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:08.692690    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdac8917_90cc_4c89_8e2e_3dccfc77a3bc.slice/crio-d994287d393000a8c6e4643a60b69a99778c2f25020de84618c4fa275f00d42d WatchSource:0}: Error finding container d994287d393000a8c6e4643a60b69a99778c2f25020de84618c4fa275f00d42d: Status 404 returned error can't find the container with id d994287d393000a8c6e4643a60b69a99778c2f25020de84618c4fa275f00d42d
Sep 07 22:33:08.729758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:08.729626    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:08.729862 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.729806    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:08.729918 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:08.729873    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.729852745 +0000 UTC m=+69.572121007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:09.092769 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.092645    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8" event={"ID":"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc","Type":"ContainerStarted","Data":"d994287d393000a8c6e4643a60b69a99778c2f25020de84618c4fa275f00d42d"}
Sep 07 22:33:09.094724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.094645    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9" event={"ID":"ab1e0fd0-df4c-4232-80ff-b3c222b86ac6","Type":"ContainerStarted","Data":"b5104034047251133af9da12dff1819510026e3f55d816598fdfaa81b79a62bd"}
Sep 07 22:33:09.096364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.096302    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7" event={"ID":"438316bc-84ec-4b46-827d-ce96b271d110","Type":"ContainerStarted","Data":"535f62fe8c8d7340daad9613b92e21e2df990a07bccd639a2e7fec1aa2da1b8d"}
Sep 07 22:33:09.637511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.637467    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:09.637701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.637542    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:09.637701 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:09.637630    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 22:33:09.637812 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:09.637712    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.637687847 +0000 UTC m=+71.479956092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : secret "router-metrics-certs-default" not found
Sep 07 22:33:09.637812 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:09.637733    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.637723357 +0000 UTC m=+71.479991602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:09.738906 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:09.738861    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:09.739079 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:09.738997    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:09.739157 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:09.739078    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.73905689 +0000 UTC m=+71.581325142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:11.654018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:11.653977    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:11.654495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:11.654095    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:11.654495 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:11.654168    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.654142491 +0000 UTC m=+75.496410736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:11.654495 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:11.654216    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 22:33:11.654495 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:11.654281    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.654267247 +0000 UTC m=+75.496535493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : secret "router-metrics-certs-default" not found
Sep 07 22:33:11.754623 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:11.754583    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:11.754814 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:11.754739    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:11.754814 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:11.754810    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.754789362 +0000 UTC m=+75.597057611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:12.563933 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.563890    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:12.570868 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.570843    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9pjg2"]
Sep 07 22:33:12.575225 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.574387    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.577350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.577328    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tb8g9\""
Sep 07 22:33:12.578893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.578458    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 22:33:12.578893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.578479    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 22:33:12.578893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.578743    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 22:33:12.578893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.578779    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 22:33:12.587287 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.587223    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pjg2"]
Sep 07 22:33:12.625926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.625895    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:12.662489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.662465    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c366661-71e1-467a-84fc-357ab5ee78cb-data-volume\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.662753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.662510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.662753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.662530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.662753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.662573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c366661-71e1-467a-84fc-357ab5ee78cb-crio-socket\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.662753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.662596    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcvs\" (UniqueName: \"kubernetes.io/projected/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-api-access-8mcvs\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.763876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.763839    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c366661-71e1-467a-84fc-357ab5ee78cb-crio-socket\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.763889    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcvs\" (UniqueName: \"kubernetes.io/projected/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-api-access-8mcvs\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.763940    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c366661-71e1-467a-84fc-357ab5ee78cb-data-volume\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.763968    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c366661-71e1-467a-84fc-357ab5ee78cb-crio-socket\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.763980    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.764047    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.764336 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:12.764069    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.764336 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:12.764144    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.264109278 +0000 UTC m=+73.106377548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.764336 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:12.764203    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.764336 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:12.764254    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.264236919 +0000 UTC m=+73.106505178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.764569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.764548    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c366661-71e1-467a-84fc-357ab5ee78cb-data-volume\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.778613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.778551    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcvs\" (UniqueName: \"kubernetes.io/projected/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-api-access-8mcvs\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:12.928934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:12.928897    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:13.109881 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.109798    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g2k4w" event={"ID":"4d2b780d-b803-45c9-b9b7-a6092b85ad76","Type":"ContainerStarted","Data":"42a8d7ad4e7907e04a0bff17408ad971bec381312fe00d32c9834361d0a50c8d"}
Sep 07 22:33:13.110037 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.109915    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:33:13.111669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.111641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9" event={"ID":"ab1e0fd0-df4c-4232-80ff-b3c222b86ac6","Type":"ContainerStarted","Data":"24e358414aa732e9adb257dd7c95c7e10733a4fd8cb45ea8c69d150768d0075a"}
Sep 07 22:33:13.113324 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.113294    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7" event={"ID":"438316bc-84ec-4b46-827d-ce96b271d110","Type":"ContainerStarted","Data":"668ddeeb5a49d850f3ddfb8083c9e28eaede67d5d78fa374f74c0e66c9e0517e"}
Sep 07 22:33:13.114857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.114835    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7772" event={"ID":"c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2","Type":"ContainerStarted","Data":"cf4d40b75808a4d513e08162c0e0fcf6419e5812ae7ab6670a4ef7b04e6553b8"}
Sep 07 22:33:13.116310 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.116287    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8" event={"ID":"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc","Type":"ContainerStarted","Data":"05c60dac69fa2255cbe1aa27c02b49d157f84d720ed7c0c7a688ece38caf392d"}
Sep 07 22:33:13.124041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.123994    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g2k4w" podStartSLOduration=66.584092569 podStartE2EDuration="1m12.123978582s" totalImagesPullingTime="5.539886013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:13.123323714 +0000 UTC m=+72.965591981" watchObservedRunningTime="2026-09-07 22:33:13.123978582 +0000 UTC m=+72.966246850"
Sep 07 22:33:13.137426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.137381    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-p46w7" podStartSLOduration=2.209117477 podStartE2EDuration="6.137365756s" totalImagesPullingTime="3.928248279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:13.13696185 +0000 UTC m=+72.979230118" watchObservedRunningTime="2026-09-07 22:33:13.137365756 +0000 UTC m=+72.979634024"
Sep 07 22:33:13.151418 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.151267    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xltb9" podStartSLOduration=2.129404108 podStartE2EDuration="6.151248098s" totalImagesPullingTime="4.02184399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:13.150033103 +0000 UTC m=+72.992301372" watchObservedRunningTime="2026-09-07 22:33:13.151248098 +0000 UTC m=+72.993516370"
Sep 07 22:33:13.170653 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.170600    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8" podStartSLOduration=1.2841796030000001 podStartE2EDuration="5.17058083s" totalImagesPullingTime="3.886401227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:13.169392192 +0000 UTC m=+73.011660460" watchObservedRunningTime="2026-09-07 22:33:13.17058083 +0000 UTC m=+73.012849097"
Sep 07 22:33:13.192728 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.192668    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c7772" podStartSLOduration=66.53551125 podStartE2EDuration="1m12.192646485s" totalImagesPullingTime="5.657135235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:13.19180722 +0000 UTC m=+73.034075485" watchObservedRunningTime="2026-09-07 22:33:13.192646485 +0000 UTC m=+73.034914756"
Sep 07 22:33:13.219332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.219298    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:13.269679 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.269637    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:13.269868 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.269691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:13.269868 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:13.269779    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:13.269868 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:13.269801    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.269868 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:13.269857    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.26983667 +0000 UTC m=+74.112104932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:13.270086 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:13.269879    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.269870155 +0000 UTC m=+74.112138401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.288373 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.288335    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:13.359995 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.359916    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:13.455797 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.455760    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:13.509928 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:13.509893    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.276038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.275989    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:14.276038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.276041    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:14.276522 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:14.276106    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.276522 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:14.276168    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.276522 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:14.276202    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.276183465 +0000 UTC m=+76.118451709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.276522 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:14.276250    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.276231984 +0000 UTC m=+76.118500253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.411639 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.411597    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.505192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.505162    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.527148 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.527079    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz776_b0676c7e-17b2-4596-b67e-9b5d54da77d4/dns-node-resolver/0.log"
Sep 07 22:33:14.605305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.605276    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.795625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.795559    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-96snr"]
Sep 07 22:33:14.799539 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.799524    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.801801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.801779    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 07 22:33:14.801913 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.801826    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 07 22:33:14.802081 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.802066    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 07 22:33:14.802927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.802908    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 07 22:33:14.803005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.802941    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ttj9r\""
Sep 07 22:33:14.805521 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.805500    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-96snr"]
Sep 07 22:33:14.827060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.827039    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.876226 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.876203    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:14.881291 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.881274    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-key\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.881353 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.881300    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xz2\" (UniqueName: \"kubernetes.io/projected/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-kube-api-access-z6xz2\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.881353 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.881336    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-cabundle\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.981690 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.981667    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-key\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.981759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.981695    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xz2\" (UniqueName: \"kubernetes.io/projected/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-kube-api-access-z6xz2\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.981759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.981720    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-cabundle\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.982364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.982347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-cabundle\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.983932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.983909    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-signing-key\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:14.989515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:14.989486    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xz2\" (UniqueName: \"kubernetes.io/projected/8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6-kube-api-access-z6xz2\") pod \"service-ca-75d99c484-96snr\" (UID: \"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6\") " pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:15.108884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.108824    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-96snr"
Sep 07 22:33:15.375640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.375556    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-96snr"]
Sep 07 22:33:15.378271 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:15.378244    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0e30ec_d3d3_4599_85b9_a3bee1cba4c6.slice/crio-7bb79f4ea6efa44fefe238a1fa4ae61c80d95a2713d8c98b005046765375696d WatchSource:0}: Error finding container 7bb79f4ea6efa44fefe238a1fa4ae61c80d95a2713d8c98b005046765375696d: Status 404 returned error can't find the container with id 7bb79f4ea6efa44fefe238a1fa4ae61c80d95a2713d8c98b005046765375696d
Sep 07 22:33:15.686010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.685929    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:15.686010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.685974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:15.686225 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:15.686085    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 22:33:15.686225 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:15.686102    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.686086066 +0000 UTC m=+83.528354314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:15.686225 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:15.686154    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.686122947 +0000 UTC m=+83.528391196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : secret "router-metrics-certs-default" not found
Sep 07 22:33:15.708621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.708589    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:15.727444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.727422    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mwfht_75cb1275-a6c3-40bf-9c65-6a2128a40564/node-ca/0.log"
Sep 07 22:33:15.786606 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:15.786570    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:15.786754 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:15.786709    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:15.786795 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:15.786770    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.786755128 +0000 UTC m=+83.629023386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:16.128119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:16.128086    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k457m_6ec8a08b-16e8-4f9f-89a0-75331e4edb80/serve-healthcheck-canary/0.log"
Sep 07 22:33:16.130400 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:16.130369    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-96snr" event={"ID":"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6","Type":"ContainerStarted","Data":"7bb79f4ea6efa44fefe238a1fa4ae61c80d95a2713d8c98b005046765375696d"}
Sep 07 22:33:16.289744 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:16.289703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:16.289911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:16.289757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:16.289911 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:16.289833    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.289911 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:16.289903    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.28988497 +0000 UTC m=+80.132153218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.290085 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:16.289836    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:16.290085 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:16.290003    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.28998225 +0000 UTC m=+80.132250512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:18.137260 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.137220    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-96snr" event={"ID":"8a0e30ec-d3d3-4599-85b9-a3bee1cba4c6","Type":"ContainerStarted","Data":"59a74c1b0bd8738e4d3ab9bd76d4394c60344e0f56ada33b56499ca3115e4532"}
Sep 07 22:33:18.153568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.153510    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-96snr" podStartSLOduration=2.394794446 podStartE2EDuration="4.153495317s" totalImagesPullingTime="1.758700871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:18.15267076 +0000 UTC m=+77.994939031" watchObservedRunningTime="2026-09-07 22:33:18.153495317 +0000 UTC m=+77.995763584"
Sep 07 22:33:18.569107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.569068    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:18.669548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.669507    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:18.729423 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.729339    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:18.787776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.787737    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:18.941173 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:18.938647    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:19.040078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:19.040045    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:19.255500 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:19.255380    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:20.062111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:20.062074    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:20.325518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:20.325413    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:20.325518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:20.325458    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:20.326032 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:20.325532    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.326032 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:20.325558    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:20.326032 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:20.325591    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.325576013 +0000 UTC m=+88.167844258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.326032 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:20.325625    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.325609561 +0000 UTC m=+88.167877811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:20.657186 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:20.657083    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:20.858293 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:20.858264    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:21.053808 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:21.053763    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:21.461048 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:21.461016    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:21.652165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:21.652102    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:22.058223 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:22.058188    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:22.159554 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:22.159523    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:22.260333 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:22.260293    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:22.859422 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:22.859390    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.053704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.053670    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.110002 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.109926    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.254680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.254631    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.456046 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.456008    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.658305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.658269    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:23.755922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.755829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:23.756095 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.755946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:23.756095 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:23.756047    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle podName:2094faa6-7b9d-4f4d-b65a-105f8a62e214 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:39.756019863 +0000 UTC m=+99.598288122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle") pod "router-default-5789d6f94b-r48qj" (UID: "2094faa6-7b9d-4f4d-b65a-105f8a62e214") : configmap references non-existent config key: service-ca.crt
Sep 07 22:33:23.758364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.758343    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2094faa6-7b9d-4f4d-b65a-105f8a62e214-metrics-certs\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:23.856872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:23.856826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:23.857063 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:23.856969    2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 22:33:23.857063 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:23.857044    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert podName:3af3b600-3fac-4e9f-b577-24d9db07bf35 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:39.85702697 +0000 UTC m=+99.699295214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f7qqt" (UID: "3af3b600-3fac-4e9f-b577-24d9db07bf35") : secret "networking-console-plugin-cert" not found
Sep 07 22:33:24.270054 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:24.270016    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:24.749472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:24.749441    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:24.862684 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:24.862651    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:25.455884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:25.455848    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:25.657958 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:25.657925    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:25.758323 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:25.758218    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:25.857976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:25.857940    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:26.323254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:26.323222    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:27.053084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:27.053054    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:27.278667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:27.278634    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:27.453510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:27.453479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:27.654167 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:27.654139    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:28.390433 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:28.390404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:28.390795 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:28.390438    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:28.390795 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:28.390542    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:28.390795 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:28.390602    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls podName:9c366661-71e1-467a-84fc-357ab5ee78cb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:44.39058638 +0000 UTC m=+104.232854625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pjg2" (UID: "9c366661-71e1-467a-84fc-357ab5ee78cb") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:28.390975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:28.390955    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c366661-71e1-467a-84fc-357ab5ee78cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:28.655596 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:28.655522    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:28.859518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:28.859482    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:29.051702 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:29.051677    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:29.265082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:29.265054    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:31.774254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:31.774213    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:31.876735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:31.876686    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:37.351294 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.351259    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"]
Sep 07 22:33:37.355610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.355586    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.359332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.359312    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 22:33:37.360581 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.360567    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 07 22:33:37.360646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.360573    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 22:33:37.361466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.361447    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 22:33:37.367944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.367921    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"]
Sep 07 22:33:37.425942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.425914    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:37.453012 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.452984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjgg\" (UniqueName: \"kubernetes.io/projected/1a709740-2f31-4829-8a23-be9399d74b09-kube-api-access-fdjgg\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.453142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.453040    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a709740-2f31-4829-8a23-be9399d74b09-tmp\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.453142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.453105    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a709740-2f31-4829-8a23-be9399d74b09-klusterlet-config\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.518437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.518406    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:37.554406 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.554381    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjgg\" (UniqueName: \"kubernetes.io/projected/1a709740-2f31-4829-8a23-be9399d74b09-kube-api-access-fdjgg\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.554496 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.554437    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a709740-2f31-4829-8a23-be9399d74b09-tmp\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.554496 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.554463    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a709740-2f31-4829-8a23-be9399d74b09-klusterlet-config\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.554830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.554814    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a709740-2f31-4829-8a23-be9399d74b09-tmp\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.556837 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.556812    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a709740-2f31-4829-8a23-be9399d74b09-klusterlet-config\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.562603 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.562584    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjgg\" (UniqueName: \"kubernetes.io/projected/1a709740-2f31-4829-8a23-be9399d74b09-kube-api-access-fdjgg\") pod \"klusterlet-addon-workmgr-7468fd98c-skq5f\" (UID: \"1a709740-2f31-4829-8a23-be9399d74b09\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.664444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.664386    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:37.928655 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:37.928584    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"]
Sep 07 22:33:37.931144 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:37.931106    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a709740_2f31_4829_8a23_be9399d74b09.slice/crio-bbc472b17de6cd796a221deaee264b77aea03853ad7a7175e23577cfd674547f WatchSource:0}: Error finding container bbc472b17de6cd796a221deaee264b77aea03853ad7a7175e23577cfd674547f: Status 404 returned error can't find the container with id bbc472b17de6cd796a221deaee264b77aea03853ad7a7175e23577cfd674547f
Sep 07 22:33:38.197936 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.197902    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f" event={"ID":"1a709740-2f31-4829-8a23-be9399d74b09","Type":"ContainerStarted","Data":"bbc472b17de6cd796a221deaee264b77aea03853ad7a7175e23577cfd674547f"}
Sep 07 22:33:38.360268 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.360239    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:38.362489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.362467    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b74b79f4-9aac-4c4d-998c-cd09b9e02b53-metrics-tls\") pod \"dns-default-svkjt\" (UID: \"b74b79f4-9aac-4c4d-998c-cd09b9e02b53\") " pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:38.664255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.664191    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:33:38.673047 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.673025    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:38.774981 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.774955    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:38.827039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.827014    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:38.942706 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:38.942662    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-svkjt"]
Sep 07 22:33:38.948430 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:38.948399    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74b79f4_9aac_4c4d_998c_cd09b9e02b53.slice/crio-130e9fa28d72d3b8a432c88b68f7abe0a0a545c79f5cdc2b4e2d33b1409cd7d3 WatchSource:0}: Error finding container 130e9fa28d72d3b8a432c88b68f7abe0a0a545c79f5cdc2b4e2d33b1409cd7d3: Status 404 returned error can't find the container with id 130e9fa28d72d3b8a432c88b68f7abe0a0a545c79f5cdc2b4e2d33b1409cd7d3
Sep 07 22:33:39.203192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.203098    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svkjt" event={"ID":"b74b79f4-9aac-4c4d-998c-cd09b9e02b53","Type":"ContainerStarted","Data":"130e9fa28d72d3b8a432c88b68f7abe0a0a545c79f5cdc2b4e2d33b1409cd7d3"}
Sep 07 22:33:39.770934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.770893    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:39.771741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.771713    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2094faa6-7b9d-4f4d-b65a-105f8a62e214-service-ca-bundle\") pod \"router-default-5789d6f94b-r48qj\" (UID: \"2094faa6-7b9d-4f4d-b65a-105f8a62e214\") " pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:39.808265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.808239    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:39.872060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.872033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:39.875155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.875112    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af3b600-3fac-4e9f-b577-24d9db07bf35-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f7qqt\" (UID: \"3af3b600-3fac-4e9f-b577-24d9db07bf35\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:39.907176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:39.907153    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt"
Sep 07 22:33:42.449678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:42.449653    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f7qqt"]
Sep 07 22:33:42.452469 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:42.452434    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af3b600_3fac_4e9f_b577_24d9db07bf35.slice/crio-d798a122a725b992f186491095b31803ccec3ca0e6e8fea5e96e9e6b27582fc1 WatchSource:0}: Error finding container d798a122a725b992f186491095b31803ccec3ca0e6e8fea5e96e9e6b27582fc1: Status 404 returned error can't find the container with id d798a122a725b992f186491095b31803ccec3ca0e6e8fea5e96e9e6b27582fc1
Sep 07 22:33:42.452960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:42.452938    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5789d6f94b-r48qj"]
Sep 07 22:33:42.455762 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:42.455737    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2094faa6_7b9d_4f4d_b65a_105f8a62e214.slice/crio-17e8a6e7dfeeabdc721de49e1150ba45c76a47b4b7e98e983f4d027753e04775 WatchSource:0}: Error finding container 17e8a6e7dfeeabdc721de49e1150ba45c76a47b4b7e98e983f4d027753e04775: Status 404 returned error can't find the container with id 17e8a6e7dfeeabdc721de49e1150ba45c76a47b4b7e98e983f4d027753e04775
Sep 07 22:33:43.219143 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.219087    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt" event={"ID":"3af3b600-3fac-4e9f-b577-24d9db07bf35","Type":"ContainerStarted","Data":"d798a122a725b992f186491095b31803ccec3ca0e6e8fea5e96e9e6b27582fc1"}
Sep 07 22:33:43.220727 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.220698    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f" event={"ID":"1a709740-2f31-4829-8a23-be9399d74b09","Type":"ContainerStarted","Data":"1ddb331d887be4aef66263e885928560bd0f863f0757e20c34d3737a4b1d0356"}
Sep 07 22:33:43.220927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.220855    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:43.222559 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.222534    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f"
Sep 07 22:33:43.222653 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.222581    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5789d6f94b-r48qj" event={"ID":"2094faa6-7b9d-4f4d-b65a-105f8a62e214","Type":"ContainerStarted","Data":"abed764811d2fd9f7e04bdf26dac2bd739b0803cb3f65370be9b08216e85a000"}
Sep 07 22:33:43.222653 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.222612    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5789d6f94b-r48qj" event={"ID":"2094faa6-7b9d-4f4d-b65a-105f8a62e214","Type":"ContainerStarted","Data":"17e8a6e7dfeeabdc721de49e1150ba45c76a47b4b7e98e983f4d027753e04775"}
Sep 07 22:33:43.224645 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.224620    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svkjt" event={"ID":"b74b79f4-9aac-4c4d-998c-cd09b9e02b53","Type":"ContainerStarted","Data":"3b4d783b13e60444bd798003a2127ada3a7980250b7ae05447b3e9069929dd06"}
Sep 07 22:33:43.224743 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.224670    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svkjt" event={"ID":"b74b79f4-9aac-4c4d-998c-cd09b9e02b53","Type":"ContainerStarted","Data":"da87deaabf36bf9a52c47a6d01ca32991ed85c4618e61289a6967c68f80d96f5"}
Sep 07 22:33:43.224815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.224758    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:43.238185 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.238137    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7468fd98c-skq5f" podStartSLOduration=1.9768656930000001 podStartE2EDuration="6.23810833s" totalImagesPullingTime="4.261242637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:43.23661914 +0000 UTC m=+103.078887407" watchObservedRunningTime="2026-09-07 22:33:43.23810833 +0000 UTC m=+103.080376598"
Sep 07 22:33:43.253401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.253353    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5789d6f94b-r48qj" podStartSLOduration=36.253334271 podStartE2EDuration="36.253334271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:43.252236075 +0000 UTC m=+103.094504344" watchObservedRunningTime="2026-09-07 22:33:43.253334271 +0000 UTC m=+103.095602538"
Sep 07 22:33:43.284250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.284202    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-svkjt" podStartSLOduration=66.055487256 podStartE2EDuration="1m9.284183893s" totalImagesPullingTime="3.228696637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:43.283726879 +0000 UTC m=+103.125995144" watchObservedRunningTime="2026-09-07 22:33:43.284183893 +0000 UTC m=+103.126452163"
Sep 07 22:33:43.350470 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.350442    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:43.523173 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.523082    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:43.532774 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.532742    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"]
Sep 07 22:33:43.568234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.568202    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"]
Sep 07 22:33:43.568385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.568300    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:43.570744 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.570718    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 07 22:33:43.570744 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.570735    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8sdvw\""
Sep 07 22:33:43.701066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.701032    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-2gm7z\" (UID: \"ddb208d4-ebf4-4c10-83da-711d9efdd09a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:43.802396 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.802322    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-2gm7z\" (UID: \"ddb208d4-ebf4-4c10-83da-711d9efdd09a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:43.802518 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:43.802461    2550 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 22:33:43.802564 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:43.802523    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates podName:ddb208d4-ebf4-4c10-83da-711d9efdd09a nodeName:}" failed. No retries permitted until 2026-09-07 22:33:44.302509284 +0000 UTC m=+104.144777529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-2gm7z" (UID: "ddb208d4-ebf4-4c10-83da-711d9efdd09a") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 22:33:43.809072 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.809050    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:43.811671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.811651    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:43.881810 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:43.881785    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:44.124252 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.124190    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g2k4w"
Sep 07 22:33:44.228247 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.228227    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:44.229650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.229628    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5789d6f94b-r48qj"
Sep 07 22:33:44.306642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.306614    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-2gm7z\" (UID: \"ddb208d4-ebf4-4c10-83da-711d9efdd09a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:44.308927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.308900    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddb208d4-ebf4-4c10-83da-711d9efdd09a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-2gm7z\" (UID: \"ddb208d4-ebf4-4c10-83da-711d9efdd09a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:44.407686 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.407628    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:44.409594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.409578    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c366661-71e1-467a-84fc-357ab5ee78cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pjg2\" (UID: \"9c366661-71e1-467a-84fc-357ab5ee78cb\") " pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:44.463955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.463932    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pjg2"
Sep 07 22:33:44.480088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.480065    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:44.761282 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.761261    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pjg2"]
Sep 07 22:33:44.763608 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:44.763586    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c366661_71e1_467a_84fc_357ab5ee78cb.slice/crio-203fdc0498167147d2ec8a1a9a437c81ae7a4ab80d85c30aa31c94b1bce576a5 WatchSource:0}: Error finding container 203fdc0498167147d2ec8a1a9a437c81ae7a4ab80d85c30aa31c94b1bce576a5: Status 404 returned error can't find the container with id 203fdc0498167147d2ec8a1a9a437c81ae7a4ab80d85c30aa31c94b1bce576a5
Sep 07 22:33:44.782676 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:44.782583    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"]
Sep 07 22:33:44.784814 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:44.784793    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb208d4_ebf4_4c10_83da_711d9efdd09a.slice/crio-1c78b0e55f04a388c089d6b0abf08511511d2e533801a7d71da00e6fc98bcc49 WatchSource:0}: Error finding container 1c78b0e55f04a388c089d6b0abf08511511d2e533801a7d71da00e6fc98bcc49: Status 404 returned error can't find the container with id 1c78b0e55f04a388c089d6b0abf08511511d2e533801a7d71da00e6fc98bcc49
Sep 07 22:33:45.233014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.232976    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt" event={"ID":"3af3b600-3fac-4e9f-b577-24d9db07bf35","Type":"ContainerStarted","Data":"3b88f531a5c88630c1123012a9e1e0ca786bc3685714337efa009884493618c5"}
Sep 07 22:33:45.234221 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.234197    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z" event={"ID":"ddb208d4-ebf4-4c10-83da-711d9efdd09a","Type":"ContainerStarted","Data":"1c78b0e55f04a388c089d6b0abf08511511d2e533801a7d71da00e6fc98bcc49"}
Sep 07 22:33:45.235536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.235504    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pjg2" event={"ID":"9c366661-71e1-467a-84fc-357ab5ee78cb","Type":"ContainerStarted","Data":"b558ec70bfb5cfca3c4b31f1a3f44d957d4fc5fb2dbaa2f764485687d8c949b7"}
Sep 07 22:33:45.235619 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.235563    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pjg2" event={"ID":"9c366661-71e1-467a-84fc-357ab5ee78cb","Type":"ContainerStarted","Data":"203fdc0498167147d2ec8a1a9a437c81ae7a4ab80d85c30aa31c94b1bce576a5"}
Sep 07 22:33:45.248093 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.248052    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-f7qqt" podStartSLOduration=35.512648285 podStartE2EDuration="37.248039712s" totalImagesPullingTime="1.735391427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:45.247292242 +0000 UTC m=+105.089560510" watchObservedRunningTime="2026-09-07 22:33:45.248039712 +0000 UTC m=+105.090308037"
Sep 07 22:33:45.343907 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.343882    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:45.396345 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:45.396318    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:47.247602 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.247567    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pjg2" event={"ID":"9c366661-71e1-467a-84fc-357ab5ee78cb","Type":"ContainerStarted","Data":"8760c4625b3f5ae501d3503ff3bd3d5b88846e56d4fbf475a7d8c2893527946a"}
Sep 07 22:33:47.249031 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.249006    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z" event={"ID":"ddb208d4-ebf4-4c10-83da-711d9efdd09a","Type":"ContainerStarted","Data":"6d39b882f2c23c0ce3ed983a7aeb4845802f1fd6a14d7815c514280268d8c581"}
Sep 07 22:33:47.249276 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.249256    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:47.253880 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.253852    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z"
Sep 07 22:33:47.264881 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.264845    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-2gm7z" podStartSLOduration=2.399297166 podStartE2EDuration="4.264832782s" totalImagesPullingTime="1.865535616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:47.263688215 +0000 UTC m=+107.105956481" watchObservedRunningTime="2026-09-07 22:33:47.264832782 +0000 UTC m=+107.107101039"
Sep 07 22:33:47.560279 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.560188    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:47.634885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:47.634849    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:48.028519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:48.028360    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:48.254250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:48.254180    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pjg2" event={"ID":"9c366661-71e1-467a-84fc-357ab5ee78cb","Type":"ContainerStarted","Data":"16bdb1c7bd1cb6c57f48651a58307cb63f45f4897b721b131d597e555f897e88"}
Sep 07 22:33:48.271043 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:48.271000    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9pjg2" podStartSLOduration=33.326618426 podStartE2EDuration="36.270987089s" totalImagesPullingTime="2.944368663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:48.269217486 +0000 UTC m=+108.111485752" watchObservedRunningTime="2026-09-07 22:33:48.270987089 +0000 UTC m=+108.113255355"
Sep 07 22:33:50.508296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:50.508269    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:52.698663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.698626    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:52.808019 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.807987    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:52.914474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.914442    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:52.926822 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.926791    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"]
Sep 07 22:33:52.930069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.930048    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:52.932353 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.932334    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 07 22:33:52.932699 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.932674    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 22:33:52.932699 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.932694    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 07 22:33:52.932867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.932785    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 22:33:52.933794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.933776    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mx477\""
Sep 07 22:33:52.933794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.933786    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 22:33:52.941318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.941296    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"]
Sep 07 22:33:52.948338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.948319    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"]
Sep 07 22:33:52.951377 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.951333    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:52.954715 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.954538    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 07 22:33:52.954955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.954923    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2rbx8\""
Sep 07 22:33:52.955222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.955202    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 07 22:33:52.955427 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.955404    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 07 22:33:52.956049 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.956031    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p5mtm"]
Sep 07 22:33:52.960998 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.960979    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:52.961835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.961765    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"]
Sep 07 22:33:52.963960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.963944    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 22:33:52.964157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.964012    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzc2k\""
Sep 07 22:33:52.964597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.964577    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 22:33:52.964822 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.964806    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 22:33:52.996876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:52.996850    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.057478 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.057458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.063561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063531    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e85d231a-1af7-42cd-9424-e104025375bf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.063678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063565    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-sys\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.063678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063594    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-wtmp\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.063678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063624    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.063678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063652    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063753    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-metrics-client-ca\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063786    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnl9z\" (UniqueName: \"kubernetes.io/projected/90a859b0-34bb-4532-afef-3d464da64afb-kube-api-access-nnl9z\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/438607bc-c547-4eac-8656-f01b48c6c696-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.063887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063926    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.063991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064015    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064039    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-textfile\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-root\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064110    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkh9\" (UniqueName: \"kubernetes.io/projected/438607bc-c547-4eac-8656-f01b48c6c696-kube-api-access-7tkh9\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064148    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfcq\" (UniqueName: \"kubernetes.io/projected/e85d231a-1af7-42cd-9424-e104025375bf-kube-api-access-vnfcq\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.064216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.064175    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-root\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165528    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkh9\" (UniqueName: \"kubernetes.io/projected/438607bc-c547-4eac-8656-f01b48c6c696-kube-api-access-7tkh9\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfcq\" (UniqueName: \"kubernetes.io/projected/e85d231a-1af7-42cd-9424-e104025375bf-kube-api-access-vnfcq\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165581    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165608    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e85d231a-1af7-42cd-9424-e104025375bf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165632    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-sys\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165658    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-wtmp\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165712    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-metrics-client-ca\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165795    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl9z\" (UniqueName: \"kubernetes.io/projected/90a859b0-34bb-4532-afef-3d464da64afb-kube-api-access-nnl9z\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165818    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/438607bc-c547-4eac-8656-f01b48c6c696-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165841    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165877    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.170896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.165982    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.166004    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-textfile\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.166375    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-textfile\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.166454    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.166624    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls podName:e85d231a-1af7-42cd-9424-e104025375bf nodeName:}" failed. No retries permitted until 2026-09-07 22:33:53.666503515 +0000 UTC m=+113.508771766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-8sk99" (UID: "e85d231a-1af7-42cd-9424-e104025375bf") : secret "openshift-state-metrics-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.166777    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-root\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.166982    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.167001    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-sys\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.167493    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.167581    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.167625    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls podName:438607bc-c547-4eac-8656-f01b48c6c696 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:53.667609334 +0000 UTC m=+113.509877594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-mgv6d" (UID: "438607bc-c547-4eac-8656-f01b48c6c696") : secret "kube-state-metrics-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.167689    2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.167718    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls podName:90a859b0-34bb-4532-afef-3d464da64afb nodeName:}" failed. No retries permitted until 2026-09-07 22:33:53.667707673 +0000 UTC m=+113.509975922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls") pod "node-exporter-p5mtm" (UID: "90a859b0-34bb-4532-afef-3d464da64afb") : secret "node-exporter-tls" not found
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.167848    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/438607bc-c547-4eac-8656-f01b48c6c696-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.171788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.168396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.172594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.168631    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-wtmp\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.172594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.168781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90a859b0-34bb-4532-afef-3d464da64afb-metrics-client-ca\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.172594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.169403    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e85d231a-1af7-42cd-9424-e104025375bf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.177754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.177710    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkh9\" (UniqueName: \"kubernetes.io/projected/438607bc-c547-4eac-8656-f01b48c6c696-kube-api-access-7tkh9\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.178330 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.178301    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.178808 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.178699    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfcq\" (UniqueName: \"kubernetes.io/projected/e85d231a-1af7-42cd-9424-e104025375bf-kube-api-access-vnfcq\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.178808 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.178776    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.179346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.179320    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.181314 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.181294    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl9z\" (UniqueName: \"kubernetes.io/projected/90a859b0-34bb-4532-afef-3d464da64afb-kube-api-access-nnl9z\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.194679 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.194541    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.230429 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.230369    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-svkjt"
Sep 07 22:33:53.315720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.315698    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.417703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.417667    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.518369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.518290    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.670373 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.670347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:53.670488 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.670402    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.670488 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.670443    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.670599 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.670488    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 07 22:33:53.670599 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:33:53.670554    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls podName:438607bc-c547-4eac-8656-f01b48c6c696 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:54.670540345 +0000 UTC m=+114.512808590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-mgv6d" (UID: "438607bc-c547-4eac-8656-f01b48c6c696") : secret "kube-state-metrics-tls" not found
Sep 07 22:33:53.672674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.672654    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90a859b0-34bb-4532-afef-3d464da64afb-node-exporter-tls\") pod \"node-exporter-p5mtm\" (UID: \"90a859b0-34bb-4532-afef-3d464da64afb\") " pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.672888 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.672867    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e85d231a-1af7-42cd-9424-e104025375bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-8sk99\" (UID: \"e85d231a-1af7-42cd-9424-e104025375bf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.807032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.806962    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.841139 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.841111    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"
Sep 07 22:33:53.873136 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.873097    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5mtm"
Sep 07 22:33:53.885037 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:53.884995    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a859b0_34bb_4532_afef_3d464da64afb.slice/crio-13ffce8575493f3ddc71810f1f02b6b041ab27235558e4cb4278d3266162be8f WatchSource:0}: Error finding container 13ffce8575493f3ddc71810f1f02b6b041ab27235558e4cb4278d3266162be8f: Status 404 returned error can't find the container with id 13ffce8575493f3ddc71810f1f02b6b041ab27235558e4cb4278d3266162be8f
Sep 07 22:33:53.919353 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.919328    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:53.987835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:53.987789    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.115078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.115057    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99"]
Sep 07 22:33:54.117538 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:54.117513    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85d231a_1af7_42cd_9424_e104025375bf.slice/crio-f595a20ab461c693e99fffd3aa1b63fbcaf6552a6d843af1c9c1cb6df90915b8 WatchSource:0}: Error finding container f595a20ab461c693e99fffd3aa1b63fbcaf6552a6d843af1c9c1cb6df90915b8: Status 404 returned error can't find the container with id f595a20ab461c693e99fffd3aa1b63fbcaf6552a6d843af1c9c1cb6df90915b8
Sep 07 22:33:54.118041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.118023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.280090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.280053    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5mtm" event={"ID":"90a859b0-34bb-4532-afef-3d464da64afb","Type":"ContainerStarted","Data":"13ffce8575493f3ddc71810f1f02b6b041ab27235558e4cb4278d3266162be8f"}
Sep 07 22:33:54.281845 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.281822    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99" event={"ID":"e85d231a-1af7-42cd-9424-e104025375bf","Type":"ContainerStarted","Data":"ce29182717c99148a4ec399bc2a2b6b32bad38fe90b942cbfd4925c010aba713"}
Sep 07 22:33:54.281934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.281867    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99" event={"ID":"e85d231a-1af7-42cd-9424-e104025375bf","Type":"ContainerStarted","Data":"b0cdb32efe4bfb030217f1663dc609990c3c91d77a86917607107f3867af4f72"}
Sep 07 22:33:54.281934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.281885    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99" event={"ID":"e85d231a-1af7-42cd-9424-e104025375bf","Type":"ContainerStarted","Data":"f595a20ab461c693e99fffd3aa1b63fbcaf6552a6d843af1c9c1cb6df90915b8"}
Sep 07 22:33:54.641287 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.641241    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.677464 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.677424    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:54.679638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.679613    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/438607bc-c547-4eac-8656-f01b48c6c696-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mgv6d\" (UID: \"438607bc-c547-4eac-8656-f01b48c6c696\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:54.690829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.690801    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.763233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.763165    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"
Sep 07 22:33:54.792241 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.792212    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.958052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.958026    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67999497fd-669vw"]
Sep 07 22:33:54.962643 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.962616    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:54.964556 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.964533    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:54.965598 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.965581    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 22:33:54.965855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.965831    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 22:33:54.965957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.965864    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 22:33:54.965957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.965890    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qgwdd\""
Sep 07 22:33:54.966240 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.966219    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9ekfck9pea3l6\""
Sep 07 22:33:54.966332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.966277    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 22:33:54.966391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.966222    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 22:33:54.972042 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:54.972023    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67999497fd-669vw"]
Sep 07 22:33:55.080848 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.080814    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.080997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.080870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxkr\" (UniqueName: \"kubernetes.io/projected/37a2f84d-6423-4c11-a65c-935d7fceaa1e-kube-api-access-zbxkr\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.080997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.080913    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.080997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.080948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.081191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.081042    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.081191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.081086    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-grpc-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.081191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.081109    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a2f84d-6423-4c11-a65c-935d7fceaa1e-metrics-client-ca\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.081191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.081159    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.113024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.112998    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.160774 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.160716    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d"]
Sep 07 22:33:55.169502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.169477    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.182340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182308    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a2f84d-6423-4c11-a65c-935d7fceaa1e-metrics-client-ca\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182393    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182411    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxkr\" (UniqueName: \"kubernetes.io/projected/37a2f84d-6423-4c11-a65c-935d7fceaa1e-kube-api-access-zbxkr\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182457    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182510    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.182622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.182550    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-grpc-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.183317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.183269    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37a2f84d-6423-4c11-a65c-935d7fceaa1e-metrics-client-ca\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.185241 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.185217    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.185423 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.185391    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.185543 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.185512    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.185683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.185659    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.186070 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.186045    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-grpc-tls\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.186321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.186300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37a2f84d-6423-4c11-a65c-935d7fceaa1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.189575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.189551    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxkr\" (UniqueName: \"kubernetes.io/projected/37a2f84d-6423-4c11-a65c-935d7fceaa1e-kube-api-access-zbxkr\") pod \"thanos-querier-67999497fd-669vw\" (UID: \"37a2f84d-6423-4c11-a65c-935d7fceaa1e\") " pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.197057 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:55.197036    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438607bc_c547_4eac_8656_f01b48c6c696.slice/crio-9d4603f64bff579fbfe20038068ebd0cfb8413ffe4c57abd2f69eb420588f9da WatchSource:0}: Error finding container 9d4603f64bff579fbfe20038068ebd0cfb8413ffe4c57abd2f69eb420588f9da: Status 404 returned error can't find the container with id 9d4603f64bff579fbfe20038068ebd0cfb8413ffe4c57abd2f69eb420588f9da
Sep 07 22:33:55.282388 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.282362    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:55.286605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.286585    2550 generic.go:356] "Generic (PLEG): container finished" podID="90a859b0-34bb-4532-afef-3d464da64afb" containerID="3503e4f1c1025a8829e5c8c7ad845ecde0de4bcc40856e7af2dbdb9686342e78" exitCode=0
Sep 07 22:33:55.286704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.286681    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5mtm" event={"ID":"90a859b0-34bb-4532-afef-3d464da64afb","Type":"ContainerDied","Data":"3503e4f1c1025a8829e5c8c7ad845ecde0de4bcc40856e7af2dbdb9686342e78"}
Sep 07 22:33:55.287858 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.287840    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d" event={"ID":"438607bc-c547-4eac-8656-f01b48c6c696","Type":"ContainerStarted","Data":"9d4603f64bff579fbfe20038068ebd0cfb8413ffe4c57abd2f69eb420588f9da"}
Sep 07 22:33:55.317831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.317802    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.545347 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.545315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.563178 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.563154    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67999497fd-669vw"]
Sep 07 22:33:55.565603 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:55.565578    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2f84d_6423_4c11_a65c_935d7fceaa1e.slice/crio-7dc3d828e3f2c4b87c7e9322fa6ebc3fd5d9c02ced3f09b41165fa54f354b261 WatchSource:0}: Error finding container 7dc3d828e3f2c4b87c7e9322fa6ebc3fd5d9c02ced3f09b41165fa54f354b261: Status 404 returned error can't find the container with id 7dc3d828e3f2c4b87c7e9322fa6ebc3fd5d9c02ced3f09b41165fa54f354b261
Sep 07 22:33:55.729212 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.729179    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.829635 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.829606    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.933108 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.933081    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:55.983893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:55.983816    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:56.057650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.057618    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:56.113431 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.113404    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:56.295867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.295741    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5mtm" event={"ID":"90a859b0-34bb-4532-afef-3d464da64afb","Type":"ContainerStarted","Data":"b997c2e7717171d5ef210b2e361fd20d0bd4ad349d584ed50e88738ea666b134"}
Sep 07 22:33:56.295867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.295808    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5mtm" event={"ID":"90a859b0-34bb-4532-afef-3d464da64afb","Type":"ContainerStarted","Data":"278d2d4374186b4886de2fe94f7be5f7657ed7688d0699312651562361cac6d1"}
Sep 07 22:33:56.299248 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.298469    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99" event={"ID":"e85d231a-1af7-42cd-9424-e104025375bf","Type":"ContainerStarted","Data":"08c1fede78b77b5fd0a690ebcab6ed1536456e074df82c4dea2a67852361dc46"}
Sep 07 22:33:56.300073 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.300042    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"7dc3d828e3f2c4b87c7e9322fa6ebc3fd5d9c02ced3f09b41165fa54f354b261"}
Sep 07 22:33:56.312502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.312455    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p5mtm" podStartSLOduration=3.19420011 podStartE2EDuration="4.312442062s" totalImagesPullingTime="1.052028178s" totalInitContainerRuntime="66.213774ms" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:56.311473952 +0000 UTC m=+116.153742220" watchObservedRunningTime="2026-09-07 22:33:56.312442062 +0000 UTC m=+116.154710319"
Sep 07 22:33:56.327726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.327679    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-8sk99" podStartSLOduration=3.324012059 podStartE2EDuration="4.327668529s" totalImagesPullingTime="1.00365647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:56.326881721 +0000 UTC m=+116.169149988" watchObservedRunningTime="2026-09-07 22:33:56.327668529 +0000 UTC m=+116.169936800"
Sep 07 22:33:56.330782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.330725    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:56.514420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:56.512550    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.113290 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.113244    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.216169 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.216119    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.275991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.275953    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.306831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.306785    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d" event={"ID":"438607bc-c547-4eac-8656-f01b48c6c696","Type":"ContainerStarted","Data":"ca3159da901fdfb3389e8db1ddac5fbc585d70b09c05a459b4d483b15e5efa05"}
Sep 07 22:33:57.306978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.306867    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d" event={"ID":"438607bc-c547-4eac-8656-f01b48c6c696","Type":"ContainerStarted","Data":"fa644ad736668f575a8f3a8cc6d5afc3ddfce8e62bdafa2dfbdfea304d67e897"}
Sep 07 22:33:57.306978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.306897    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d" event={"ID":"438607bc-c547-4eac-8656-f01b48c6c696","Type":"ContainerStarted","Data":"ef27d0470e37c8b2806d274ea840a831a66744093d897afd942a006beb7c1cb7"}
Sep 07 22:33:57.323273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.323219    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mgv6d" podStartSLOduration=3.936143749 podStartE2EDuration="5.323202397s" totalImagesPullingTime="1.387058648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:57.322700186 +0000 UTC m=+117.164968455" watchObservedRunningTime="2026-09-07 22:33:57.323202397 +0000 UTC m=+117.165470665"
Sep 07 22:33:57.716928 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.716367    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"]
Sep 07 22:33:57.717567 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.717521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.720994 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.720933    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.723355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.723332    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 07 22:33:57.723355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.723350    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r8xpz\""
Sep 07 22:33:57.725909 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.725856    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"]
Sep 07 22:33:57.803121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.803091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2584\" (UniqueName: \"kubernetes.io/projected/41af9e32-f8b9-48b0-841f-d64624e079ce-kube-api-access-l2584\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.803265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.803162    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41af9e32-f8b9-48b0-841f-d64624e079ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.809220 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.809195    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.889905 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.889846    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:57.903829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.903805    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41af9e32-f8b9-48b0-841f-d64624e079ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.903912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.903895    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2584\" (UniqueName: \"kubernetes.io/projected/41af9e32-f8b9-48b0-841f-d64624e079ce-kube-api-access-l2584\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.906065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.906045    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41af9e32-f8b9-48b0-841f-d64624e079ce-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.910736 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.910715    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2584\" (UniqueName: \"kubernetes.io/projected/41af9e32-f8b9-48b0-841f-d64624e079ce-kube-api-access-l2584\") pod \"monitoring-plugin-85bbbbbf4-p7ppd\" (UID: \"41af9e32-f8b9-48b0-841f-d64624e079ce\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:57.945165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:57.945141    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:58.043832 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:58.043813    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:33:58.309854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:58.309824    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"]
Sep 07 22:33:58.313200 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:33:58.313172    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41af9e32_f8b9_48b0_841f_d64624e079ce.slice/crio-d3eab20a01a406e7bbcfc8da50730684ae003b9a8dba783fd9b5e2bda9921687 WatchSource:0}: Error finding container d3eab20a01a406e7bbcfc8da50730684ae003b9a8dba783fd9b5e2bda9921687: Status 404 returned error can't find the container with id d3eab20a01a406e7bbcfc8da50730684ae003b9a8dba783fd9b5e2bda9921687
Sep 07 22:33:58.313962 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:58.313936    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"5bd7387c50ab287c746115c3d3e2c226f224714e4a45dab9dfcf4eff6000c574"}
Sep 07 22:33:58.314026 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:58.314011    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"a13abac6c1bc30e923af51a4533e1cd1627b26ed9fcfb68ab3b3d1b42d2be576"}
Sep 07 22:33:58.314063 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:58.314040    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"42240250fa5861bb76265465ff3478fda7bd35240a37663ec213cea74b32db3c"}
Sep 07 22:33:59.203286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.203255    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:59.321668 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.321629    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"259b888453dcc0198a7417f4521f3b6405f0380e9e14ad9fa199c847b2b7e4b6"}
Sep 07 22:33:59.322087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.321709    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"39459efb608cbdf3a5f4174f42fcfa7bb5ed60b294c12927191ba783d2fbac4c"}
Sep 07 22:33:59.322087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.321755    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" event={"ID":"37a2f84d-6423-4c11-a65c-935d7fceaa1e","Type":"ContainerStarted","Data":"a0d57ce91f9f4f2451ef0be8660c3708039b42337fdea5e67d2fcd89392f3326"}
Sep 07 22:33:59.322087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.321807    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:33:59.331005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.328893    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd" event={"ID":"41af9e32-f8b9-48b0-841f-d64624e079ce","Type":"ContainerStarted","Data":"d3eab20a01a406e7bbcfc8da50730684ae003b9a8dba783fd9b5e2bda9921687"}
Sep 07 22:33:59.345601 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.345557    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67999497fd-669vw" podStartSLOduration=2.453060978 podStartE2EDuration="5.345542636s" totalImagesPullingTime="2.892481658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:59.343984861 +0000 UTC m=+119.186253128" watchObservedRunningTime="2026-09-07 22:33:59.345542636 +0000 UTC m=+119.187810903"
Sep 07 22:33:59.565199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.565103    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:59.619475 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.619435    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:59.688932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.688910    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:33:59.790109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:33:59.790073    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:00.334406 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:00.334363    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd" event={"ID":"41af9e32-f8b9-48b0-841f-d64624e079ce","Type":"ContainerStarted","Data":"864560df0c8059f1de056fba9e9b9f29f65403c293a6c4233eea3bf67c134225"}
Sep 07 22:34:00.334854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:00.334761    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:34:00.339682 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:00.339659    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd"
Sep 07 22:34:00.348500 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:00.348453    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-p7ppd" podStartSLOduration=2.016180894 podStartE2EDuration="3.348438538s" totalImagesPullingTime="1.332257644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:00.348068913 +0000 UTC m=+120.190337171" watchObservedRunningTime="2026-09-07 22:34:00.348438538 +0000 UTC m=+120.190706805"
Sep 07 22:34:01.131656 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:01.131620    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:01.231939 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:01.231910    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:01.770843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:01.770795    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:01.816429 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:01.816397    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:03.338999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:03.338950    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:03.429109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:03.429078    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:04.297060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.297026    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:04.300336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.300319    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.302817 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.302784    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 22:34:04.302954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.302834    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 22:34:04.302954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.302899    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 22:34:04.303792 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.303775    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 22:34:04.304171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.304156    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lhcs8\""
Sep 07 22:34:04.304277 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.304260    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 22:34:04.304356 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.304307    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 22:34:04.304449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.304431    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 22:34:04.309010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.308990    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:04.454089 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454066    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.454473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454093    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.454473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbdz\" (UniqueName: \"kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.454473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454159    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.454473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454264    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.454473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.454312    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555283 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555223    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555283 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555248    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555283 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555267    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbdz\" (UniqueName: \"kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555390    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.555985 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.555963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.556111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.556089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.556199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.556089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.557662 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.557643    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.557752 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.557649    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.565007 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.564987    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbdz\" (UniqueName: \"kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz\") pod \"console-87bc7f88c-hnb2z\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") " pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.610107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.610085    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:04.676764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.676738    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:04.726568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.726543    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:04.878193 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:04.878169    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:04.880621 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:34:04.880596    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede51430_3f69_45c3_8d8b_460a1ce6de51.slice/crio-46dc90d85d71107e293dff437428d5de3399a2798045611a23b29580eb5d6115 WatchSource:0}: Error finding container 46dc90d85d71107e293dff437428d5de3399a2798045611a23b29580eb5d6115: Status 404 returned error can't find the container with id 46dc90d85d71107e293dff437428d5de3399a2798045611a23b29580eb5d6115
Sep 07 22:34:05.339992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:05.339965    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67999497fd-669vw"
Sep 07 22:34:05.354121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:05.354092    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bc7f88c-hnb2z" event={"ID":"ede51430-3f69-45c3-8d8b-460a1ce6de51","Type":"ContainerStarted","Data":"46dc90d85d71107e293dff437428d5de3399a2798045611a23b29580eb5d6115"}
Sep 07 22:34:08.367935 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:08.367851    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bc7f88c-hnb2z" event={"ID":"ede51430-3f69-45c3-8d8b-460a1ce6de51","Type":"ContainerStarted","Data":"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"}
Sep 07 22:34:08.931852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:08.931823    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.030973 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.030941    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.079793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.079766    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.139176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.139144    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.193547 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.193522    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.437584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.437558    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:09.488346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:09.488278    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:10.504309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.504278    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:34:10.506516 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.506492    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99661a91-1abb-48c6-99f0-97ee822cab04-metrics-certs\") pod \"network-metrics-daemon-hds5x\" (UID: \"99661a91-1abb-48c6-99f0-97ee822cab04\") " pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:34:10.603794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.603765    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:34:10.611621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.611602    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hds5x"
Sep 07 22:34:10.876467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.876367    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-87bc7f88c-hnb2z" podStartSLOduration=3.702759226 podStartE2EDuration="6.876353716s" totalImagesPullingTime="3.17359449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:08.38358509 +0000 UTC m=+128.225853357" watchObservedRunningTime="2026-09-07 22:34:10.876353716 +0000 UTC m=+130.718621982"
Sep 07 22:34:10.877827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:10.877793    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hds5x"]
Sep 07 22:34:10.880383 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:34:10.880357    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99661a91_1abb_48c6_99f0_97ee822cab04.slice/crio-87d3af7b74eb5277c767e4cf7bdf7c94079dc4fcd73dfdb17c20599094f1936f WatchSource:0}: Error finding container 87d3af7b74eb5277c767e4cf7bdf7c94079dc4fcd73dfdb17c20599094f1936f: Status 404 returned error can't find the container with id 87d3af7b74eb5277c767e4cf7bdf7c94079dc4fcd73dfdb17c20599094f1936f
Sep 07 22:34:11.379465 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:11.379434    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hds5x" event={"ID":"99661a91-1abb-48c6-99f0-97ee822cab04","Type":"ContainerStarted","Data":"87d3af7b74eb5277c767e4cf7bdf7c94079dc4fcd73dfdb17c20599094f1936f"}
Sep 07 22:34:12.384918 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.384827    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hds5x" event={"ID":"99661a91-1abb-48c6-99f0-97ee822cab04","Type":"ContainerStarted","Data":"00eebb864138b452a953aba517b8bb21b60a832eac3513a66a7bfd2740d396e7"}
Sep 07 22:34:12.384918 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.384881    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hds5x" event={"ID":"99661a91-1abb-48c6-99f0-97ee822cab04","Type":"ContainerStarted","Data":"12673ff19636551ca8d76a90d53b6442d340efc66a33878026d9267af981737b"}
Sep 07 22:34:12.403180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.403110    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hds5x" podStartSLOduration=131.29368165 podStartE2EDuration="2m12.403097042s" totalImagesPullingTime="1.109415392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:12.402384972 +0000 UTC m=+132.244653240" watchObservedRunningTime="2026-09-07 22:34:12.403097042 +0000 UTC m=+132.245365309"
Sep 07 22:34:12.503452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.503424    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:12.609179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.609148    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:12.703329 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:12.703300    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:14.142031 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:14.141989    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:14.229110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:14.229025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:14.610512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:14.610482    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:14.610670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:14.610527    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:14.615221 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:14.615199    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:15.100788 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:15.100741    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:15.197857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:15.197831    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:15.399688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:15.399614    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:16.314566 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:16.314532    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:16.414980 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:16.414945    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:16.776010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:16.775969    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:16.836340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:16.836308    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:18.409158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.409114    2550 generic.go:356] "Generic (PLEG): container finished" podID="fdac8917-90cc-4c89-8e2e-3dccfc77a3bc" containerID="05c60dac69fa2255cbe1aa27c02b49d157f84d720ed7c0c7a688ece38caf392d" exitCode=0
Sep 07 22:34:18.409441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.409159    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8" event={"ID":"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc","Type":"ContainerDied","Data":"05c60dac69fa2255cbe1aa27c02b49d157f84d720ed7c0c7a688ece38caf392d"}
Sep 07 22:34:18.409489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.409439    2550 scope.go:117] "RemoveContainer" containerID="05c60dac69fa2255cbe1aa27c02b49d157f84d720ed7c0c7a688ece38caf392d"
Sep 07 22:34:18.507518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.507488    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:18.557685 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.557653    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:18.943867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:18.943837    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:19.044237 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:19.044203    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:19.415043 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:19.414964    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t92v8" event={"ID":"fdac8917-90cc-4c89-8e2e-3dccfc77a3bc","Type":"ContainerStarted","Data":"94c77c3d0a75ee28027f5b0110744fb3a7280a5b585f6c539032536350fbc8a2"}
Sep 07 22:34:22.681347 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:22.681306    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:22.841723 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:22.841690    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:22.941737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:22.941704    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:25.746320 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:25.746283    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:25.800948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:25.800920    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:31.770626 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:31.770590    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:31.821906 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:31.821873    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:34:47.701064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.700994    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-87bc7f88c-hnb2z" podUID="ede51430-3f69-45c3-8d8b-460a1ce6de51" containerName="console" containerID="cri-o://40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a" gracePeriod=15
Sep 07 22:34:47.942407 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.942382    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:47.958275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958194    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.958404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958312    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdbdz\" (UniqueName: \"kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.958404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958338    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.958404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958369    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.958404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958399    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.958621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.958445    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert\") pod \"ede51430-3f69-45c3-8d8b-460a1ce6de51\" (UID: \"ede51430-3f69-45c3-8d8b-460a1ce6de51\") "
Sep 07 22:34:47.959646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.959388    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config" (OuterVolumeSpecName: "console-config") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:47.960067 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.960032    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca" (OuterVolumeSpecName: "service-ca") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:47.960421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.960373    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:47.961208 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.961156    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:47.961302 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.961274    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz" (OuterVolumeSpecName: "kube-api-access-fdbdz") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "kube-api-access-fdbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:47.961508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:47.961480    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ede51430-3f69-45c3-8d8b-460a1ce6de51" (UID: "ede51430-3f69-45c3-8d8b-460a1ce6de51"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:48.059722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059695    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbdz\" (UniqueName: \"kubernetes.io/projected/ede51430-3f69-45c3-8d8b-460a1ce6de51-kube-api-access-fdbdz\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.059722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059720    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.059876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059729    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.059876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059738    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede51430-3f69-45c3-8d8b-460a1ce6de51-oauth-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.059876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059747    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.059876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.059756    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede51430-3f69-45c3-8d8b-460a1ce6de51-console-oauth-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:34:48.523844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.523810    2550 generic.go:356] "Generic (PLEG): container finished" podID="ede51430-3f69-45c3-8d8b-460a1ce6de51" containerID="40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a" exitCode=0
Sep 07 22:34:48.524011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.523856    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87bc7f88c-hnb2z"
Sep 07 22:34:48.524011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.523876    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bc7f88c-hnb2z" event={"ID":"ede51430-3f69-45c3-8d8b-460a1ce6de51","Type":"ContainerDied","Data":"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"}
Sep 07 22:34:48.524011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.523916    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87bc7f88c-hnb2z" event={"ID":"ede51430-3f69-45c3-8d8b-460a1ce6de51","Type":"ContainerDied","Data":"46dc90d85d71107e293dff437428d5de3399a2798045611a23b29580eb5d6115"}
Sep 07 22:34:48.524011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.523936    2550 scope.go:117] "RemoveContainer" containerID="40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"
Sep 07 22:34:48.533246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.533223    2550 scope.go:117] "RemoveContainer" containerID="40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"
Sep 07 22:34:48.533482 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:34:48.533462    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a\": container with ID starting with 40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a not found: ID does not exist" containerID="40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"
Sep 07 22:34:48.533536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.533490    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a"} err="failed to get container status \"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a\": rpc error: code = NotFound desc = could not find container \"40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a\": container with ID starting with 40055ce0ff657ea1b649d06583b6c94626ce05b8c16ad6e143b4a8962f634e9a not found: ID does not exist"
Sep 07 22:34:48.543228 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.543205    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:48.547634 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.547603    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87bc7f88c-hnb2z"]
Sep 07 22:34:48.794577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:34:48.794502    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede51430-3f69-45c3-8d8b-460a1ce6de51" path="/var/lib/kubelet/pods/ede51430-3f69-45c3-8d8b-460a1ce6de51/volumes"
Sep 07 22:35:09.023700 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:09.023655    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:09.071736 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:09.071697    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:14.685376 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:14.685330    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:20.902569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:20.902521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:20.949864 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:20.949836    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:31.781159 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:31.781109    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:31.880754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:31.880718    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:34.200756 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:34.200723    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:54.631981 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.631941    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:35:54.632491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.632396    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede51430-3f69-45c3-8d8b-460a1ce6de51" containerName="console"
Sep 07 22:35:54.632491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.632420    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede51430-3f69-45c3-8d8b-460a1ce6de51" containerName="console"
Sep 07 22:35:54.632602 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.632497    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede51430-3f69-45c3-8d8b-460a1ce6de51" containerName="console"
Sep 07 22:35:54.635548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.635517    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.637908 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.637884    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 22:35:54.638055 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.638038    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 22:35:54.639056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639020    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 22:35:54.639056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639028    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lhcs8\""
Sep 07 22:35:54.639255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639097    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 22:35:54.639255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639182    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 22:35:54.639419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639402    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 22:35:54.639519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.639500    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 22:35:54.643444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.643401    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 22:35:54.645183 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.645160    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:35:54.648857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.648833    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.648957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.648867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxw6w\" (UniqueName: \"kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.648957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.648890    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.649054 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.648957    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.649054 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.649005    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.649054 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.649035    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.649233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.649062    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.720258 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.720227    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:54.750355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750316    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750464 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750366    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750464 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750400    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750464 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750463    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxw6w\" (UniqueName: \"kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750490    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.750580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.750538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.751196 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.751163    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.751318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.751211    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.751318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.751240    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.751318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.751301    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.752877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.752858    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.752980 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.752909    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.758007 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.757986    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxw6w\" (UniqueName: \"kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w\") pod \"console-5f7f687dff-nqq6l\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") " pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:54.769142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.769104    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:54.945703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:54.945684    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:35:55.030072 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.030046    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:55.075279 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.075254    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:55.214185 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.214092    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:35:55.217770 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:35:55.217741    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97557405_ef80_46f4_92c7_972abcc3a353.slice/crio-3a857e10467b031fd5e0368469a715e0b021328fa89c9d4ef4ae7dfa87b01259 WatchSource:0}: Error finding container 3a857e10467b031fd5e0368469a715e0b021328fa89c9d4ef4ae7dfa87b01259: Status 404 returned error can't find the container with id 3a857e10467b031fd5e0368469a715e0b021328fa89c9d4ef4ae7dfa87b01259
Sep 07 22:35:55.783875 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.783840    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f687dff-nqq6l" event={"ID":"97557405-ef80-46f4-92c7-972abcc3a353","Type":"ContainerStarted","Data":"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"}
Sep 07 22:35:55.784342 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.783896    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f687dff-nqq6l" event={"ID":"97557405-ef80-46f4-92c7-972abcc3a353","Type":"ContainerStarted","Data":"3a857e10467b031fd5e0368469a715e0b021328fa89c9d4ef4ae7dfa87b01259"}
Sep 07 22:35:55.802828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.802772    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f7f687dff-nqq6l" podStartSLOduration=1.8027539510000001 podStartE2EDuration="1.802753951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:35:55.800448414 +0000 UTC m=+235.642716681" watchObservedRunningTime="2026-09-07 22:35:55.802753951 +0000 UTC m=+235.645022221"
Sep 07 22:35:55.836267 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.836225    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:55.886286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:55.886258    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:58.501006 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:58.500972    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:58.600444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:58.600416    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:35:59.065373 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.065343    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"]
Sep 07 22:35:59.068825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.068805    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.071205 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.071180    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:35:59.071324 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.071230    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:35:59.072287 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.072268    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:35:59.076435 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.076414    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"]
Sep 07 22:35:59.080588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.080565    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.080691 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.080591    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wh4l\" (UniqueName: \"kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.080691 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.080612    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.181393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.181364    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.181498 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.181395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wh4l\" (UniqueName: \"kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.181498 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.181418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.181706 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.181687    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.181746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.181714    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.189565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.189544    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wh4l\" (UniqueName: \"kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.379781 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.379729    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:35:59.649164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.649119    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"]
Sep 07 22:35:59.651644 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:35:59.651614    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377e3e9c_14f5_47f0_a82c_2aa12cacfeb4.slice/crio-078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf WatchSource:0}: Error finding container 078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf: Status 404 returned error can't find the container with id 078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf
Sep 07 22:35:59.800827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:35:59.800802    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll" event={"ID":"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4","Type":"ContainerStarted","Data":"078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf"}
Sep 07 22:36:04.946469 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:04.946431    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:36:04.947028 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:04.946499    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:36:04.952939 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:04.952912    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:36:05.829252 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:05.829214    2550 generic.go:356] "Generic (PLEG): container finished" podID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerID="f170657d24ea6e8beb57ed9dac4eb16f6dc1652a61d95173b7806d89c332b8f6" exitCode=0
Sep 07 22:36:05.829430 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:05.829295    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll" event={"ID":"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4","Type":"ContainerDied","Data":"f170657d24ea6e8beb57ed9dac4eb16f6dc1652a61d95173b7806d89c332b8f6"}
Sep 07 22:36:05.833351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:05.833330    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:36:05.978741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:05.978712    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:06.027996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:06.027971    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:06.079182 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:06.079159    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:06.127873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:06.127823    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:07.839121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:07.839056    2550 generic.go:356] "Generic (PLEG): container finished" podID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerID="6eeaf27e11afca8c13527f05dcc1cffd382384cf1e2c25b038d588c50e8c6596" exitCode=0
Sep 07 22:36:07.839441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:07.839140    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll" event={"ID":"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4","Type":"ContainerDied","Data":"6eeaf27e11afca8c13527f05dcc1cffd382384cf1e2c25b038d588c50e8c6596"}
Sep 07 22:36:08.624141 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:08.624086    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:08.675547 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:08.675511    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:14.869216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:14.869180    2550 generic.go:356] "Generic (PLEG): container finished" podID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerID="be42edfef7745b80a9715372288cba7b0aed2fe5e42049df60a578d06f0a4829" exitCode=0
Sep 07 22:36:14.869582 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:14.869254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll" event={"ID":"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4","Type":"ContainerDied","Data":"be42edfef7745b80a9715372288cba7b0aed2fe5e42049df60a578d06f0a4829"}
Sep 07 22:36:15.995589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:15.995562    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:36:16.115381 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.115357    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wh4l\" (UniqueName: \"kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l\") pod \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") "
Sep 07 22:36:16.115521 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.115422    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle\") pod \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") "
Sep 07 22:36:16.115521 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.115444    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util\") pod \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\" (UID: \"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4\") "
Sep 07 22:36:16.116069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.116039    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle" (OuterVolumeSpecName: "bundle") pod "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" (UID: "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:16.117434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.117406    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l" (OuterVolumeSpecName: "kube-api-access-9wh4l") pod "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" (UID: "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4"). InnerVolumeSpecName "kube-api-access-9wh4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:36:16.119532 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.119508    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util" (OuterVolumeSpecName: "util") pod "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" (UID: "377e3e9c-14f5-47f0-a82c-2aa12cacfeb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:16.215829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.215808    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:36:16.215829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.215828    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-util\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:36:16.215957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.215837    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wh4l\" (UniqueName: \"kubernetes.io/projected/377e3e9c-14f5-47f0-a82c-2aa12cacfeb4-kube-api-access-9wh4l\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:36:16.877818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.877793    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll"
Sep 07 22:36:16.877818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.877796    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mqlll" event={"ID":"377e3e9c-14f5-47f0-a82c-2aa12cacfeb4","Type":"ContainerDied","Data":"078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf"}
Sep 07 22:36:16.878019 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:16.877837    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078e318f2f4e05b6618fa851ec07458784015da113fc84a04a5450812c511eaf"
Sep 07 22:36:19.834454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:19.834423    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:21.200768 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.200732    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:21.754707 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.754675    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"]
Sep 07 22:36:21.755024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755001    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="util"
Sep 07 22:36:21.755024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755021    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="util"
Sep 07 22:36:21.755211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755042    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="pull"
Sep 07 22:36:21.755211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755051    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="pull"
Sep 07 22:36:21.755211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755062    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="extract"
Sep 07 22:36:21.755211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755071    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="extract"
Sep 07 22:36:21.755211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.755147    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="377e3e9c-14f5-47f0-a82c-2aa12cacfeb4" containerName="extract"
Sep 07 22:36:21.759965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.759941    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.762410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.762389    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:36:21.762523 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.762461    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 07 22:36:21.762523 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.762489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ff58v\""
Sep 07 22:36:21.770865 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.770844    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"]
Sep 07 22:36:21.852534 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.852510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ggn\" (UniqueName: \"kubernetes.io/projected/57ded4b5-5d91-4130-8fe7-29fc4524324c-kube-api-access-d5ggn\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.852639 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.852539    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57ded4b5-5d91-4130-8fe7-29fc4524324c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.953164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.953136    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ggn\" (UniqueName: \"kubernetes.io/projected/57ded4b5-5d91-4130-8fe7-29fc4524324c-kube-api-access-d5ggn\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.953164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.953168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57ded4b5-5d91-4130-8fe7-29fc4524324c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.953492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.953476    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57ded4b5-5d91-4130-8fe7-29fc4524324c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:21.962622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:21.962598    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ggn\" (UniqueName: \"kubernetes.io/projected/57ded4b5-5d91-4130-8fe7-29fc4524324c-kube-api-access-d5ggn\") pod \"cert-manager-operator-controller-manager-55c444bf86-888l6\" (UID: \"57ded4b5-5d91-4130-8fe7-29fc4524324c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:22.069966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:22.069908    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"
Sep 07 22:36:22.371747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:22.371669    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6"]
Sep 07 22:36:22.374394 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:36:22.374360    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ded4b5_5d91_4130_8fe7_29fc4524324c.slice/crio-509c3ae23c5c7ce8ac6b32c4f25471121537049de07e5479b9fa4d8a91c19b5b WatchSource:0}: Error finding container 509c3ae23c5c7ce8ac6b32c4f25471121537049de07e5479b9fa4d8a91c19b5b: Status 404 returned error can't find the container with id 509c3ae23c5c7ce8ac6b32c4f25471121537049de07e5479b9fa4d8a91c19b5b
Sep 07 22:36:22.902578 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:22.902504    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6" event={"ID":"57ded4b5-5d91-4130-8fe7-29fc4524324c","Type":"ContainerStarted","Data":"509c3ae23c5c7ce8ac6b32c4f25471121537049de07e5479b9fa4d8a91c19b5b"}
Sep 07 22:36:23.424911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:23.424879    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:23.479147 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:23.479096    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:24.912529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:24.912441    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6" event={"ID":"57ded4b5-5d91-4130-8fe7-29fc4524324c","Type":"ContainerStarted","Data":"5d0da7a5bcbfd0ec2b82ec55983fd68d9096b0e09accf8b1b28adaf37aa2aa3c"}
Sep 07 22:36:24.933589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:24.933537    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-888l6" podStartSLOduration=1.708797272 podStartE2EDuration="3.933517827s" totalImagesPullingTime="2.224720555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:36:24.931607557 +0000 UTC m=+264.773875827" watchObservedRunningTime="2026-09-07 22:36:24.933517827 +0000 UTC m=+264.775786099"
Sep 07 22:36:25.298424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:25.298378    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:25.403595 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:25.403536    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:25.513812 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:25.513779    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:25.631065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:25.630980    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:26.624710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.624672    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wgmdf"]
Sep 07 22:36:26.646351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.646323    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.646517 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.646327    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wgmdf"]
Sep 07 22:36:26.648938 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.648915    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 22:36:26.649984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.649954    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 22:36:26.650083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.650013    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-x792r\""
Sep 07 22:36:26.690366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.690339    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.690455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.690384    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4nz\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-kube-api-access-rp4nz\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.791261 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.791232    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4nz\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-kube-api-access-rp4nz\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.791355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.791313    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.805828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.805798    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.805926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.805870    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4nz\" (UniqueName: \"kubernetes.io/projected/5ed0846a-6b17-49c1-9d97-da4679d61f73-kube-api-access-rp4nz\") pod \"cert-manager-webhook-6486ffd997-wgmdf\" (UID: \"5ed0846a-6b17-49c1-9d97-da4679d61f73\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:26.966363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:26.966333    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:27.243777 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:27.243750    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wgmdf"]
Sep 07 22:36:27.246200 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:36:27.246174    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ed0846a_6b17_49c1_9d97_da4679d61f73.slice/crio-2624e829a189988596c93e0432bf49a526c1a75c3d8dcab7265d26d7190dc64a WatchSource:0}: Error finding container 2624e829a189988596c93e0432bf49a526c1a75c3d8dcab7265d26d7190dc64a: Status 404 returned error can't find the container with id 2624e829a189988596c93e0432bf49a526c1a75c3d8dcab7265d26d7190dc64a
Sep 07 22:36:27.924693 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:27.924659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf" event={"ID":"5ed0846a-6b17-49c1-9d97-da4679d61f73","Type":"ContainerStarted","Data":"2624e829a189988596c93e0432bf49a526c1a75c3d8dcab7265d26d7190dc64a"}
Sep 07 22:36:28.059410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:28.059379    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:28.249301 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:28.249269    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:30.431928 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:30.431845    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:30.537571 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:30.537524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:30.940526 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:30.939903    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf" event={"ID":"5ed0846a-6b17-49c1-9d97-da4679d61f73","Type":"ContainerStarted","Data":"70e849940dc308feea7b0c4479d909f9b341160362b5d56593ea2883cc0efb65"}
Sep 07 22:36:30.940526 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:30.939981    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:30.955344 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:30.955299    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf" podStartSLOduration=2.1823176220000002 podStartE2EDuration="4.955283402s" totalImagesPullingTime="2.77296578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:36:30.954684453 +0000 UTC m=+270.796952720" watchObservedRunningTime="2026-09-07 22:36:30.955283402 +0000 UTC m=+270.797551669"
Sep 07 22:36:35.249790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:35.249754    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:35.932273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:35.932241    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dw9nf"]
Sep 07 22:36:35.934376 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:35.934359    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:35.936577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:35.936554    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bs62r\""
Sep 07 22:36:35.943054 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:35.943025    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dw9nf"]
Sep 07 22:36:36.062974 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.062949    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-bound-sa-token\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.063086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.062978    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ttx\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-kube-api-access-t7ttx\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.163932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.163901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-bound-sa-token\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.164053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.163943    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ttx\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-kube-api-access-t7ttx\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.171737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.171710    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-bound-sa-token\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.171848 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.171827    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ttx\" (UniqueName: \"kubernetes.io/projected/1247e148-641a-42e8-8fb4-29529e57a3da-kube-api-access-t7ttx\") pod \"cert-manager-55b4d954d5-dw9nf\" (UID: \"1247e148-641a-42e8-8fb4-29529e57a3da\") " pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.243672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.243640    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dw9nf"
Sep 07 22:36:36.509469 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.509443    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dw9nf"]
Sep 07 22:36:36.511627 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:36:36.511598    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1247e148_641a_42e8_8fb4_29529e57a3da.slice/crio-e0d610fea3b9cfc1555249dbdba4e8ea26f0ced3360d6593fd9b2ffa4f2a5c51 WatchSource:0}: Error finding container e0d610fea3b9cfc1555249dbdba4e8ea26f0ced3360d6593fd9b2ffa4f2a5c51: Status 404 returned error can't find the container with id e0d610fea3b9cfc1555249dbdba4e8ea26f0ced3360d6593fd9b2ffa4f2a5c51
Sep 07 22:36:36.947097 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.947066    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wgmdf"
Sep 07 22:36:36.966351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.966318    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dw9nf" event={"ID":"1247e148-641a-42e8-8fb4-29529e57a3da","Type":"ContainerStarted","Data":"b8fdcaa7e6cb91110e36e52b90d6b044655b4edb39f44514ecf47b6921f3d9bf"}
Sep 07 22:36:36.966506 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.966363    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dw9nf" event={"ID":"1247e148-641a-42e8-8fb4-29529e57a3da","Type":"ContainerStarted","Data":"e0d610fea3b9cfc1555249dbdba4e8ea26f0ced3360d6593fd9b2ffa4f2a5c51"}
Sep 07 22:36:36.982569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:36.982515    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dw9nf" podStartSLOduration=1.982496722 podStartE2EDuration="1.982496722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:36:36.981140962 +0000 UTC m=+276.823409222" watchObservedRunningTime="2026-09-07 22:36:36.982496722 +0000 UTC m=+276.824764991"
Sep 07 22:36:38.169843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:38.169811    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:38.271335 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:38.271264    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:48.034657 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:48.034628    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:48.134705 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:48.134674    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:53.870922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:53.870885    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:53.978057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:53.978023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:55.975474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:55.975438    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:57.276954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:57.276922    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:36:57.377631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:36:57.377597    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:00.569273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:00.569247    2550 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 22:37:08.646474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:08.646435    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:08.743536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:08.743484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.255893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.255862    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.461691 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.461662    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.561295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.561210    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.677386 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.677350    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"]
Sep 07 22:37:09.680008 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.679983    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.682772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.682736    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 22:37:09.682772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.682769    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdhlh\""
Sep 07 22:37:09.682965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.682792    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 22:37:09.682965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.682771    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 22:37:09.682965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.682770    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 22:37:09.694832 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.694811    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"]
Sep 07 22:37:09.781169 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.781110    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.815051 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.814962    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.815051 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.815003    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.815262 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.815070    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.815262 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.815111    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8f9\" (UniqueName: \"kubernetes.io/projected/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-kube-api-access-tz8f9\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.815262 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.815165    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.881265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.881193    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:09.915988 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.915955    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.916263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.916006    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8f9\" (UniqueName: \"kubernetes.io/projected/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-kube-api-access-tz8f9\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.916263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.916029    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.916263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.916057    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.916263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.916077    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.916593 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.916564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.918653 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.918631    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.918800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.918782    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.918923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.918907    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.924480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.924461    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8f9\" (UniqueName: \"kubernetes.io/projected/7da6ed5b-70f7-4893-9061-abf8eb5cde9b-kube-api-access-tz8f9\") pod \"opendatahub-operator-controller-manager-7d648d744d-btp2q\" (UID: \"7da6ed5b-70f7-4893-9061-abf8eb5cde9b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.990841 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.990812    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:09.994663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:09.994637    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:10.083211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.083145    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:10.184367 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.184324    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:10.258460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.258425    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:10.269612 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.269452    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"]
Sep 07 22:37:10.272179 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:10.272148    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da6ed5b_70f7_4893_9061_abf8eb5cde9b.slice/crio-4d72ad2ed19b8bb43425e1ac067985fb72745b3cc570433d880095b7a7193529 WatchSource:0}: Error finding container 4d72ad2ed19b8bb43425e1ac067985fb72745b3cc570433d880095b7a7193529: Status 404 returned error can't find the container with id 4d72ad2ed19b8bb43425e1ac067985fb72745b3cc570433d880095b7a7193529
Sep 07 22:37:10.274046 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.274026    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:37:10.341001 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:10.340880    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:11.123393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:11.123354    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q" event={"ID":"7da6ed5b-70f7-4893-9061-abf8eb5cde9b","Type":"ContainerStarted","Data":"4d72ad2ed19b8bb43425e1ac067985fb72745b3cc570433d880095b7a7193529"}
Sep 07 22:37:14.139986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:14.139946    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q" event={"ID":"7da6ed5b-70f7-4893-9061-abf8eb5cde9b","Type":"ContainerStarted","Data":"658ab7a5b11554c154dfcfbc9dd6e1eb7cbc6e103f111a6d77d36975dab6ec92"}
Sep 07 22:37:15.146413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:15.146369    2550 generic.go:356] "Generic (PLEG): container finished" podID="7da6ed5b-70f7-4893-9061-abf8eb5cde9b" containerID="658ab7a5b11554c154dfcfbc9dd6e1eb7cbc6e103f111a6d77d36975dab6ec92" exitCode=0
Sep 07 22:37:15.146800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:15.146455    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q" event={"ID":"7da6ed5b-70f7-4893-9061-abf8eb5cde9b","Type":"ContainerDied","Data":"658ab7a5b11554c154dfcfbc9dd6e1eb7cbc6e103f111a6d77d36975dab6ec92"}
Sep 07 22:37:16.153064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:16.153021    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q" event={"ID":"7da6ed5b-70f7-4893-9061-abf8eb5cde9b","Type":"ContainerStarted","Data":"00ebf3d940eda7e569e4c7336645e60012924ea1f09859bc0bd50aaf48aa7038"}
Sep 07 22:37:16.153487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:16.153160    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:16.174118 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:16.174063    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q" podStartSLOduration=2.773327385 podStartE2EDuration="7.174043128s" totalImagesPullingTime="3.330147827s" totalInitContainerRuntime="1.070567916s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:16.171704796 +0000 UTC m=+316.013973062" watchObservedRunningTime="2026-09-07 22:37:16.174043128 +0000 UTC m=+316.016311397"
Sep 07 22:37:21.656906 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:21.656878    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:21.755671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:21.755639    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:27.159637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:27.159603    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-btp2q"
Sep 07 22:37:28.876654 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:28.876625    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:29.962017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.961981    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"]
Sep 07 22:37:29.967754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.967731    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:29.971150 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.971104    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 22:37:29.971150 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.971105    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f7cgl\""
Sep 07 22:37:29.971310 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.971105    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 22:37:29.976758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:29.976734    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"]
Sep 07 22:37:30.057270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.057242    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tmp\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.057411 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.057281    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.057411 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.057303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcm4l\" (UniqueName: \"kubernetes.io/projected/a26da7d1-cee2-4b67-adcc-8f599c11e65e-kube-api-access-vcm4l\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.101496 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.101467    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:30.157991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.157963    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tmp\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.158108 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.158000    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.158108 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.158023    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcm4l\" (UniqueName: \"kubernetes.io/projected/a26da7d1-cee2-4b67-adcc-8f599c11e65e-kube-api-access-vcm4l\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.160292 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.160270    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tmp\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.160420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.160401    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26da7d1-cee2-4b67-adcc-8f599c11e65e-tls-certs\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.164919 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.164895    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcm4l\" (UniqueName: \"kubernetes.io/projected/a26da7d1-cee2-4b67-adcc-8f599c11e65e-kube-api-access-vcm4l\") pod \"kube-auth-proxy-68c66dfc89-p9qb4\" (UID: \"a26da7d1-cee2-4b67-adcc-8f599c11e65e\") " pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.277925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.277824    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"
Sep 07 22:37:30.559176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.559147    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4"]
Sep 07 22:37:30.561287 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:30.561261    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26da7d1_cee2_4b67_adcc_8f599c11e65e.slice/crio-8a9c441d0a4cc4be3844c2dd4bb5bb0d4a9cc612e992b5781f8d2fe0fda5a597 WatchSource:0}: Error finding container 8a9c441d0a4cc4be3844c2dd4bb5bb0d4a9cc612e992b5781f8d2fe0fda5a597: Status 404 returned error can't find the container with id 8a9c441d0a4cc4be3844c2dd4bb5bb0d4a9cc612e992b5781f8d2fe0fda5a597
Sep 07 22:37:30.979678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.979648    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:30.987171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.987118    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:30.990111 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.990093    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:30.992503 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.992463    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 07 22:37:30.992618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.992510    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 07 22:37:30.992618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.992541    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qcbl7\""
Sep 07 22:37:30.992618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.992610    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 22:37:30.992836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:30.992821    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 07 22:37:31.003234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.003216    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:31.037699 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.037680    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:31.065912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.065891    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066002 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.065934    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066002 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.065952    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.066021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.066057    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.066082    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqww\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.066163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.066149    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167742 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167637    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167742 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167697    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167742 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167723    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167915 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167915 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.167915 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.167817    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqww\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.168596 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.168563    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.170580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.170552    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.170873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.170849    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.170969 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.170929    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.171142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.171111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.175401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.175375    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqww\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.175540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.175516    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xzjmc\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.221804 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.221763    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4" event={"ID":"a26da7d1-cee2-4b67-adcc-8f599c11e65e","Type":"ContainerStarted","Data":"8a9c441d0a4cc4be3844c2dd4bb5bb0d4a9cc612e992b5781f8d2fe0fda5a597"}
Sep 07 22:37:31.299555 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.299466    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:31.619114 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.619029    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:31.621918 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:31.621878    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd72d9f_da64_4d6c_b13f_e6ba4a9de0ff.slice/crio-789d2d8979eb78a752584c2babea8c0e5cfb828e3c846158099be89fcbabb09b WatchSource:0}: Error finding container 789d2d8979eb78a752584c2babea8c0e5cfb828e3c846158099be89fcbabb09b: Status 404 returned error can't find the container with id 789d2d8979eb78a752584c2babea8c0e5cfb828e3c846158099be89fcbabb09b
Sep 07 22:37:31.911723 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:31.911630    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:32.228656 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:32.228617    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" event={"ID":"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff","Type":"ContainerStarted","Data":"789d2d8979eb78a752584c2babea8c0e5cfb828e3c846158099be89fcbabb09b"}
Sep 07 22:37:34.239817 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:34.239780    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4" event={"ID":"a26da7d1-cee2-4b67-adcc-8f599c11e65e","Type":"ContainerStarted","Data":"9dd5e549a6aef83e9f1ee3e0763a1e50b9e4f41bd36f3de54fa32aa482592b90"}
Sep 07 22:37:34.256983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:34.256935    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68c66dfc89-p9qb4" podStartSLOduration=2.532089247 podStartE2EDuration="5.256918861s" totalImagesPullingTime="2.724829614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:34.255738204 +0000 UTC m=+334.098006472" watchObservedRunningTime="2026-09-07 22:37:34.256918861 +0000 UTC m=+334.099187156"
Sep 07 22:37:35.132782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:35.132745    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:37:35.132889 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:35.132817    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:37:35.246508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:35.246465    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" event={"ID":"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff","Type":"ContainerStarted","Data":"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"}
Sep 07 22:37:35.246862 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:35.246551    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:35.268445 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:35.268391    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" podStartSLOduration=1.760936125 podStartE2EDuration="5.268377228s" totalImagesPullingTime="3.507441103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:35.266221914 +0000 UTC m=+335.108490183" watchObservedRunningTime="2026-09-07 22:37:35.268377228 +0000 UTC m=+335.110645495"
Sep 07 22:37:36.015611 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.015575    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:36.115402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.115369    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:36.252354 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.252319    2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xzjmc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:37:36.252709 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.252371    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:37:36.605610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.605570    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:36.777947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.777912    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:36.880521 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:36.880437    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:39.251644 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.251609    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:39.845099 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.845068    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtn7q"]
Sep 07 22:37:39.848510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.848487    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:39.851092 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.851063    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 07 22:37:39.851216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.851097    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h6wp5\""
Sep 07 22:37:39.856402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.856074    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtn7q"]
Sep 07 22:37:39.900966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.900930    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:39.946942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.946916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:39.947078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:39.946961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxmk\" (UniqueName: \"kubernetes.io/projected/e1bd5a87-03ac-47d2-b817-a351191470d1-kube-api-access-vbxmk\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.002908 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.002883    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:40.048182 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.048147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.048317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.048216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxmk\" (UniqueName: \"kubernetes.io/projected/e1bd5a87-03ac-47d2-b817-a351191470d1-kube-api-access-vbxmk\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.048317 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:40.048268    2550 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 07 22:37:40.048403 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:40.048325    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert podName:e1bd5a87-03ac-47d2-b817-a351191470d1 nodeName:}" failed. No retries permitted until 2026-09-07 22:37:40.548308892 +0000 UTC m=+340.390577138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert") pod "odh-model-controller-7998c67dc4-wtn7q" (UID: "e1bd5a87-03ac-47d2-b817-a351191470d1") : secret "odh-model-controller-webhook-cert" not found
Sep 07 22:37:40.060391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.060365    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxmk\" (UniqueName: \"kubernetes.io/projected/e1bd5a87-03ac-47d2-b817-a351191470d1-kube-api-access-vbxmk\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.103008 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.102941    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:40.240280 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.240245    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:40.423322 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.423232    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tw2sg"]
Sep 07 22:37:40.426528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.426508    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.428923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.428902    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 07 22:37:40.433256 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.433233    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:40.438253 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.438228    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tw2sg"]
Sep 07 22:37:40.553122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.553083    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqhf\" (UniqueName: \"kubernetes.io/projected/b22bcf05-840c-49b5-bcd5-e896a4419835-kube-api-access-swqhf\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.553321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.553145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.553321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.553214    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.555501 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.555482    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bd5a87-03ac-47d2-b817-a351191470d1-cert\") pod \"odh-model-controller-7998c67dc4-wtn7q\" (UID: \"e1bd5a87-03ac-47d2-b817-a351191470d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:40.653985 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.653942    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.654193 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.654059    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqhf\" (UniqueName: \"kubernetes.io/projected/b22bcf05-840c-49b5-bcd5-e896a4419835-kube-api-access-swqhf\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.654193 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:40.654073    2550 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 07 22:37:40.654193 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:40.654152    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs podName:b22bcf05-840c-49b5-bcd5-e896a4419835 nodeName:}" failed. No retries permitted until 2026-09-07 22:37:41.15411559 +0000 UTC m=+340.996383835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs") pod "model-serving-api-54d89bd5fc-tw2sg" (UID: "b22bcf05-840c-49b5-bcd5-e896a4419835") : secret "model-serving-api-tls" not found
Sep 07 22:37:40.664594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.664558    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqhf\" (UniqueName: \"kubernetes.io/projected/b22bcf05-840c-49b5-bcd5-e896a4419835-kube-api-access-swqhf\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:40.729035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.728993    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:40.759824 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:40.759795    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:41.028457 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.028430    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtn7q"]
Sep 07 22:37:41.030180 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:41.030155    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bd5a87_03ac_47d2_b817_a351191470d1.slice/crio-10d9bed3063095522ac7dc56cdbc7913d30d8f028b7741b436d2ff6639d03b2f WatchSource:0}: Error finding container 10d9bed3063095522ac7dc56cdbc7913d30d8f028b7741b436d2ff6639d03b2f: Status 404 returned error can't find the container with id 10d9bed3063095522ac7dc56cdbc7913d30d8f028b7741b436d2ff6639d03b2f
Sep 07 22:37:41.158283 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.158249    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:41.160577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.160546    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22bcf05-840c-49b5-bcd5-e896a4419835-tls-certs\") pod \"model-serving-api-54d89bd5fc-tw2sg\" (UID: \"b22bcf05-840c-49b5-bcd5-e896a4419835\") " pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:41.274084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.274048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q" event={"ID":"e1bd5a87-03ac-47d2-b817-a351191470d1","Type":"ContainerStarted","Data":"10d9bed3063095522ac7dc56cdbc7913d30d8f028b7741b436d2ff6639d03b2f"}
Sep 07 22:37:41.338229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.338146    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:41.607448 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:41.607423    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tw2sg"]
Sep 07 22:37:41.609029 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:41.609003    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22bcf05_840c_49b5_bcd5_e896a4419835.slice/crio-46458707ee9304e2b331ea3b8f9ad47e8d66ff48af8a5c7ee43b42c2c8488e04 WatchSource:0}: Error finding container 46458707ee9304e2b331ea3b8f9ad47e8d66ff48af8a5c7ee43b42c2c8488e04: Status 404 returned error can't find the container with id 46458707ee9304e2b331ea3b8f9ad47e8d66ff48af8a5c7ee43b42c2c8488e04
Sep 07 22:37:42.283413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:42.283371    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg" event={"ID":"b22bcf05-840c-49b5-bcd5-e896a4419835","Type":"ContainerStarted","Data":"46458707ee9304e2b331ea3b8f9ad47e8d66ff48af8a5c7ee43b42c2c8488e04"}
Sep 07 22:37:44.294716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.294676    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg" event={"ID":"b22bcf05-840c-49b5-bcd5-e896a4419835","Type":"ContainerStarted","Data":"39651459b340be8c55b611169e91465e42177eaf919163e61b8b03840e1ef7ba"}
Sep 07 22:37:44.295181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.294746    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:37:44.296107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.296083    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q" event={"ID":"e1bd5a87-03ac-47d2-b817-a351191470d1","Type":"ContainerStarted","Data":"8dfe0f567360023a5bd0609c472082ff8aacba00635a14edff1a8133aa8ab52e"}
Sep 07 22:37:44.296234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.296154    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:44.309471 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.309421    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg" podStartSLOduration=1.795847596 podStartE2EDuration="4.309402628s" totalImagesPullingTime="2.513555032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:44.308750981 +0000 UTC m=+344.151019242" watchObservedRunningTime="2026-09-07 22:37:44.309402628 +0000 UTC m=+344.151670896"
Sep 07 22:37:44.325528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:44.325484    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q" podStartSLOduration=2.230423705 podStartE2EDuration="5.325469812s" totalImagesPullingTime="3.095046107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:44.32362227 +0000 UTC m=+344.165890536" watchObservedRunningTime="2026-09-07 22:37:44.325469812 +0000 UTC m=+344.167738078"
Sep 07 22:37:46.554835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.554792    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"]
Sep 07 22:37:46.558215 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.558187    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.569800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.569774    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"]
Sep 07 22:37:46.610565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610625    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610661    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610795    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.610886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.610811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrt4x\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-kube-api-access-xrt4x\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712196 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712157    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712244    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712302    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712489    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712529    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.712645 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.712556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrt4x\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-kube-api-access-xrt4x\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.713119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.713091    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.714726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.714699    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.714840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.714776    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.714840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.714794    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.714951 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.714892    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.719697 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.719677    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.719892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.719873    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrt4x\" (UniqueName: \"kubernetes.io/projected/c4420d24-b59a-4cc3-90b1-f2ed4e48d0be-kube-api-access-xrt4x\") pod \"istiod-openshift-gateway-5677ccb49b-7sbtl\" (UID: \"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:46.870003 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:46.869909    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:47.149005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.148977    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"]
Sep 07 22:37:47.150840 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:47.150811    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4420d24_b59a_4cc3_90b1_f2ed4e48d0be.slice/crio-8ee274b06451ba73bd04fdd70568b0f3e15979fea82035cd181c6d6d41f5eacb WatchSource:0}: Error finding container 8ee274b06451ba73bd04fdd70568b0f3e15979fea82035cd181c6d6d41f5eacb: Status 404 returned error can't find the container with id 8ee274b06451ba73bd04fdd70568b0f3e15979fea82035cd181c6d6d41f5eacb
Sep 07 22:37:47.154069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.154030    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:37:47.154210 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.154108    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:37:47.311871 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.311830    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl" event={"ID":"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be","Type":"ContainerStarted","Data":"786de9fa987cd3f15e5c5b38e355ff1f13353bea0af2d83ff8f1417af565b9f3"}
Sep 07 22:37:47.312048 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.311890    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl" event={"ID":"c4420d24-b59a-4cc3-90b1-f2ed4e48d0be","Type":"ContainerStarted","Data":"8ee274b06451ba73bd04fdd70568b0f3e15979fea82035cd181c6d6d41f5eacb"}
Sep 07 22:37:47.312048 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.311935    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:47.352450 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:47.352400    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl" podStartSLOduration=1.352382449 podStartE2EDuration="1.352382449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:47.350851925 +0000 UTC m=+347.193120191" watchObservedRunningTime="2026-09-07 22:37:47.352382449 +0000 UTC m=+347.194650707"
Sep 07 22:37:48.318476 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.318450    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7sbtl"
Sep 07 22:37:48.383392 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.383360    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:48.383559 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.383538    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerName="discovery" containerID="cri-o://7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29" gracePeriod=30
Sep 07 22:37:48.622934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.622911    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:48.732208 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732182    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732223    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732242    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732262    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732298    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732551 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732352    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732551 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732410    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwqww\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww\") pod \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\" (UID: \"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff\") "
Sep 07 22:37:48.732683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732626    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:37:48.732744 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.732715    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-ca-configmap\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.734671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.734629    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts" (OuterVolumeSpecName: "cacerts") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:37:48.734758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.734705    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs" (OuterVolumeSpecName: "local-certs") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:37:48.734924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.734887    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:37:48.735021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.734953    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:37:48.735021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.734964    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:37:48.735166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.735149    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww" (OuterVolumeSpecName: "kube-api-access-rwqww") pod "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" (UID: "7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff"). InnerVolumeSpecName "kube-api-access-rwqww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:37:48.833636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833607    2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-local-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.833636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833632    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqww\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-kube-api-access-rwqww\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.833787 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833646    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-csr-dns-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.833787 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833659    2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-kubeconfig\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.833787 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833667    2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-cacerts\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:48.833787 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:48.833677    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:37:49.192986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.192956    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:49.259049 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.259011    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s2564"]
Sep 07 22:37:49.259489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.259465    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerName="discovery"
Sep 07 22:37:49.259489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.259487    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerName="discovery"
Sep 07 22:37:49.259691 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.259594    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerName="discovery"
Sep 07 22:37:49.262931 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.262909    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.265630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.265595    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-64j4f\""
Sep 07 22:37:49.265630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.265602    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 07 22:37:49.275564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.275538    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s2564"]
Sep 07 22:37:49.296872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.296841    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:49.322890 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.322860    2550 generic.go:356] "Generic (PLEG): container finished" podID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" containerID="7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29" exitCode=0
Sep 07 22:37:49.323250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.322926    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"
Sep 07 22:37:49.323250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.322948    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" event={"ID":"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff","Type":"ContainerDied","Data":"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"}
Sep 07 22:37:49.323250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.323006    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc" event={"ID":"7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff","Type":"ContainerDied","Data":"789d2d8979eb78a752584c2babea8c0e5cfb828e3c846158099be89fcbabb09b"}
Sep 07 22:37:49.323250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.323036    2550 scope.go:117] "RemoveContainer" containerID="7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"
Sep 07 22:37:49.332086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.332069    2550 scope.go:117] "RemoveContainer" containerID="7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"
Sep 07 22:37:49.332388 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:49.332368    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29\": container with ID starting with 7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29 not found: ID does not exist" containerID="7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"
Sep 07 22:37:49.332459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.332394    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29"} err="failed to get container status \"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29\": rpc error: code = NotFound desc = could not find container \"7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29\": container with ID starting with 7efdeb758146a1e44863f103b632c86ae2c831104fd8639cb7dfcf9d14171a29 not found: ID does not exist"
Sep 07 22:37:49.339605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.339579    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:49.340407 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.340385    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.340452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.340430    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f497h\" (UniqueName: \"kubernetes.io/projected/429d30f9-8d79-490b-8045-139ee29171a6-kube-api-access-f497h\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.344303 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.344282    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xzjmc"]
Sep 07 22:37:49.441542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.441505    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.441693 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.441549    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f497h\" (UniqueName: \"kubernetes.io/projected/429d30f9-8d79-490b-8045-139ee29171a6-kube-api-access-f497h\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.441693 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:49.441655    2550 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 07 22:37:49.441800 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:37:49.441728    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert podName:429d30f9-8d79-490b-8045-139ee29171a6 nodeName:}" failed. No retries permitted until 2026-09-07 22:37:49.941706487 +0000 UTC m=+349.783974752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert") pod "kserve-controller-manager-d6f4b65b8-s2564" (UID: "429d30f9-8d79-490b-8045-139ee29171a6") : secret "kserve-webhook-server-cert" not found
Sep 07 22:37:49.450439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.450371    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:49.451068 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.451049    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f497h\" (UniqueName: \"kubernetes.io/projected/429d30f9-8d79-490b-8045-139ee29171a6-kube-api-access-f497h\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.739325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.739250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:49.946030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.946004    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:49.948309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:49.948286    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429d30f9-8d79-490b-8045-139ee29171a6-cert\") pod \"kserve-controller-manager-d6f4b65b8-s2564\" (UID: \"429d30f9-8d79-490b-8045-139ee29171a6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:50.176013 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.175915    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:50.475475 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.475444    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s2564"]
Sep 07 22:37:50.477005 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:37:50.476978    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429d30f9_8d79_490b_8045_139ee29171a6.slice/crio-8bf3a8b704dbe36c612022db747b5e089fae42a68dc2a432815f67eeb033634b WatchSource:0}: Error finding container 8bf3a8b704dbe36c612022db747b5e089fae42a68dc2a432815f67eeb033634b: Status 404 returned error can't find the container with id 8bf3a8b704dbe36c612022db747b5e089fae42a68dc2a432815f67eeb033634b
Sep 07 22:37:50.515105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.515053    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:50.615287 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.615267    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:50.716029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.716000    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:37:50.797309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:50.797214    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff" path="/var/lib/kubelet/pods/7dd72d9f-da64-4d6c-b13f-e6ba4a9de0ff/volumes"
Sep 07 22:37:51.337078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:51.337034    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564" event={"ID":"429d30f9-8d79-490b-8045-139ee29171a6","Type":"ContainerStarted","Data":"8bf3a8b704dbe36c612022db747b5e089fae42a68dc2a432815f67eeb033634b"}
Sep 07 22:37:53.349059 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:53.349016    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564" event={"ID":"429d30f9-8d79-490b-8045-139ee29171a6","Type":"ContainerStarted","Data":"78f07b92a34812beb6530c46a7f4319a041dbea54b15c58eca7499d6ef3c303a"}
Sep 07 22:37:53.349494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:53.349099    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:37:53.365391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:53.365349    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564" podStartSLOduration=2.085358877 podStartE2EDuration="4.365337574s" totalImagesPullingTime="2.279978697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:53.364241078 +0000 UTC m=+353.206509345" watchObservedRunningTime="2026-09-07 22:37:53.365337574 +0000 UTC m=+353.207605841"
Sep 07 22:37:55.302509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:55.302480    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtn7q"
Sep 07 22:37:55.304761 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:37:55.304739    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tw2sg"
Sep 07 22:38:24.358956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:24.358922    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s2564"
Sep 07 22:38:40.926361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:40.926321    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:40.988979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:40.988944    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:44.000449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.000410    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"]
Sep 07 22:38:44.006842 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.006816    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.010071 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.010048    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:38:44.010202 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.010067    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:38:44.010679 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.010655    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"]
Sep 07 22:38:44.010908 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.010889    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:38:44.039301 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.039275    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.039419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.039337    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.039419 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.039357    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxz2\" (UniqueName: \"kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.139784 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.139757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.139894 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.139816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.139894 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.139836    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxz2\" (UniqueName: \"kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.140214 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.140195    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.140214 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.140208    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.147519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.147494    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxz2\" (UniqueName: \"kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.191983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.191959    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"]
Sep 07 22:38:44.196014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.195998    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.205382 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.205356    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"]
Sep 07 22:38:44.241153 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.241114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.241250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.241178    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gbg\" (UniqueName: \"kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.241296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.241253    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.295438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.295391    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"]
Sep 07 22:38:44.299278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.299262    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.307693 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.307671    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"]
Sep 07 22:38:44.317313 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.317275    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:44.341774 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.341749    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.341876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.341797    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.341876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.341826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.341876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.341870    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gbg\" (UniqueName: \"kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.342009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.341896    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.342201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.342176    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.342321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.342244    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956t9\" (UniqueName: \"kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.342321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.342300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.350567 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.350544    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gbg\" (UniqueName: \"kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.444490 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.444456    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.444614 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.444506    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.444614 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.444534    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956t9\" (UniqueName: \"kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.444836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.444816    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.444876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.444860    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.452096 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.452074    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-956t9\" (UniqueName: \"kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.505794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.505768    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:44.595632 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.595606    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"]
Sep 07 22:38:44.596734 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:38:44.596710    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb689a5fe_34ee_45fc_b1c1_c371b02d444a.slice/crio-c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af WatchSource:0}: Error finding container c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af: Status 404 returned error can't find the container with id c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af
Sep 07 22:38:44.608078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.608061    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:44.774883 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.774856    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"]
Sep 07 22:38:44.786024 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:38:44.786000    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44049cfd_fa97_4d5e_b86a_d140c4362190.slice/crio-f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107 WatchSource:0}: Error finding container f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107: Status 404 returned error can't find the container with id f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107
Sep 07 22:38:44.878005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:44.877979    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"]
Sep 07 22:38:45.593454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.593412    2550 generic.go:356] "Generic (PLEG): container finished" podID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerID="1f7ea7472181958f56b7fb4efb27483161883f2f6852abae4faa521bb87a5209" exitCode=0
Sep 07 22:38:45.593895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.593495    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb" event={"ID":"44049cfd-fa97-4d5e-b86a-d140c4362190","Type":"ContainerDied","Data":"1f7ea7472181958f56b7fb4efb27483161883f2f6852abae4faa521bb87a5209"}
Sep 07 22:38:45.593895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.593546    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb" event={"ID":"44049cfd-fa97-4d5e-b86a-d140c4362190","Type":"ContainerStarted","Data":"f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107"}
Sep 07 22:38:45.595349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.595325    2550 generic.go:356] "Generic (PLEG): container finished" podID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerID="3a604e5b1557393c66bb3145f4fa9097ed3c4d4f3799848f1f230b66e0d8f263" exitCode=0
Sep 07 22:38:45.595445 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.595410    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw" event={"ID":"b689a5fe-34ee-45fc-b1c1-c371b02d444a","Type":"ContainerDied","Data":"3a604e5b1557393c66bb3145f4fa9097ed3c4d4f3799848f1f230b66e0d8f263"}
Sep 07 22:38:45.595508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.595464    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw" event={"ID":"b689a5fe-34ee-45fc-b1c1-c371b02d444a","Type":"ContainerStarted","Data":"c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af"}
Sep 07 22:38:45.597174 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.597148    2550 generic.go:356] "Generic (PLEG): container finished" podID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerID="de50d22186314082beeaef235f32ef60ff40df596f1112d08e2f3c0eff097059" exitCode=0
Sep 07 22:38:45.597282 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.597219    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj" event={"ID":"fd2dd1c5-0762-4da9-ab50-31edfc0f468a","Type":"ContainerDied","Data":"de50d22186314082beeaef235f32ef60ff40df596f1112d08e2f3c0eff097059"}
Sep 07 22:38:45.597282 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:45.597267    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj" event={"ID":"fd2dd1c5-0762-4da9-ab50-31edfc0f468a","Type":"ContainerStarted","Data":"7bb8729956c97ed33c6ca6df26cf659befb1e24c91e5aa9d32ec6302facda004"}
Sep 07 22:38:46.603942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:46.603913    2550 generic.go:356] "Generic (PLEG): container finished" podID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerID="2e744931a9d97bd7f84e45febd310f670cc49276654375bef8feb1a6d6b54783" exitCode=0
Sep 07 22:38:46.604389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:46.603992    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw" event={"ID":"b689a5fe-34ee-45fc-b1c1-c371b02d444a","Type":"ContainerDied","Data":"2e744931a9d97bd7f84e45febd310f670cc49276654375bef8feb1a6d6b54783"}
Sep 07 22:38:46.605991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:46.605972    2550 generic.go:356] "Generic (PLEG): container finished" podID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerID="2506b99a085dbed6f70e575033f8264de8673dc3957ce37e3e5503c329bd0554" exitCode=0
Sep 07 22:38:46.606082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:46.606039    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj" event={"ID":"fd2dd1c5-0762-4da9-ab50-31edfc0f468a","Type":"ContainerDied","Data":"2506b99a085dbed6f70e575033f8264de8673dc3957ce37e3e5503c329bd0554"}
Sep 07 22:38:47.613665 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.613574    2550 generic.go:356] "Generic (PLEG): container finished" podID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerID="bbbb50e602c508bcc1db3b583c1c002eae08e135f4501560f8da97c7ed7ce66a" exitCode=0
Sep 07 22:38:47.614069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.613663    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj" event={"ID":"fd2dd1c5-0762-4da9-ab50-31edfc0f468a","Type":"ContainerDied","Data":"bbbb50e602c508bcc1db3b583c1c002eae08e135f4501560f8da97c7ed7ce66a"}
Sep 07 22:38:47.615455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.615433    2550 generic.go:356] "Generic (PLEG): container finished" podID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerID="326129199f1d886adb9d6660b9f7340c4c2ea284a0270442bbf71be75aa826e8" exitCode=0
Sep 07 22:38:47.615520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.615505    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb" event={"ID":"44049cfd-fa97-4d5e-b86a-d140c4362190","Type":"ContainerDied","Data":"326129199f1d886adb9d6660b9f7340c4c2ea284a0270442bbf71be75aa826e8"}
Sep 07 22:38:47.617911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.617889    2550 generic.go:356] "Generic (PLEG): container finished" podID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerID="c22d249d75b7591a799ebadce3e3ec7837e7de32b7b63dcf4aba7c2c04b5a9fb" exitCode=0
Sep 07 22:38:47.618023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:47.617928    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw" event={"ID":"b689a5fe-34ee-45fc-b1c1-c371b02d444a","Type":"ContainerDied","Data":"c22d249d75b7591a799ebadce3e3ec7837e7de32b7b63dcf4aba7c2c04b5a9fb"}
Sep 07 22:38:48.625265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.625230    2550 generic.go:356] "Generic (PLEG): container finished" podID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerID="6e0b43f406f505b73328d5c78347bb27fb58f3742ff8ae319cb1939746be4093" exitCode=0
Sep 07 22:38:48.625623 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.625318    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb" event={"ID":"44049cfd-fa97-4d5e-b86a-d140c4362190","Type":"ContainerDied","Data":"6e0b43f406f505b73328d5c78347bb27fb58f3742ff8ae319cb1939746be4093"}
Sep 07 22:38:48.755311 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.755287    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:48.781257 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.781231    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle\") pod \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") "
Sep 07 22:38:48.781369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.781270    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util\") pod \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") "
Sep 07 22:38:48.781369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.781291    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-956t9\" (UniqueName: \"kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9\") pod \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\" (UID: \"fd2dd1c5-0762-4da9-ab50-31edfc0f468a\") "
Sep 07 22:38:48.781725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.781698    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle" (OuterVolumeSpecName: "bundle") pod "fd2dd1c5-0762-4da9-ab50-31edfc0f468a" (UID: "fd2dd1c5-0762-4da9-ab50-31edfc0f468a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.783780 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.783751    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9" (OuterVolumeSpecName: "kube-api-access-956t9") pod "fd2dd1c5-0762-4da9-ab50-31edfc0f468a" (UID: "fd2dd1c5-0762-4da9-ab50-31edfc0f468a"). InnerVolumeSpecName "kube-api-access-956t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:38:48.786638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.786610    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util" (OuterVolumeSpecName: "util") pod "fd2dd1c5-0762-4da9-ab50-31edfc0f468a" (UID: "fd2dd1c5-0762-4da9-ab50-31edfc0f468a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.795441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.795415    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:48.882546 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882475    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util\") pod \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") "
Sep 07 22:38:48.882546 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882510    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle\") pod \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") "
Sep 07 22:38:48.882741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882572    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wxz2\" (UniqueName: \"kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2\") pod \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\" (UID: \"b689a5fe-34ee-45fc-b1c1-c371b02d444a\") "
Sep 07 22:38:48.882805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882742    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.882805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882753    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-util\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.882805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.882762    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956t9\" (UniqueName: \"kubernetes.io/projected/fd2dd1c5-0762-4da9-ab50-31edfc0f468a-kube-api-access-956t9\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.883169 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.883142    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle" (OuterVolumeSpecName: "bundle") pod "b689a5fe-34ee-45fc-b1c1-c371b02d444a" (UID: "b689a5fe-34ee-45fc-b1c1-c371b02d444a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.884522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.884501    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2" (OuterVolumeSpecName: "kube-api-access-4wxz2") pod "b689a5fe-34ee-45fc-b1c1-c371b02d444a" (UID: "b689a5fe-34ee-45fc-b1c1-c371b02d444a"). InnerVolumeSpecName "kube-api-access-4wxz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:38:48.887941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.887915    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util" (OuterVolumeSpecName: "util") pod "b689a5fe-34ee-45fc-b1c1-c371b02d444a" (UID: "b689a5fe-34ee-45fc-b1c1-c371b02d444a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.984098 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.984073    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-util\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.984098 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.984095    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b689a5fe-34ee-45fc-b1c1-c371b02d444a-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.984280 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:48.984105    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxz2\" (UniqueName: \"kubernetes.io/projected/b689a5fe-34ee-45fc-b1c1-c371b02d444a-kube-api-access-4wxz2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:49.631694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.631658    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw"
Sep 07 22:38:49.631694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.631679    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053k6jdw" event={"ID":"b689a5fe-34ee-45fc-b1c1-c371b02d444a","Type":"ContainerDied","Data":"c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af"}
Sep 07 22:38:49.632157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.631738    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96aaeb732532a81db249e4f41d0d8d8340cfc8cad581cd847447c1d8e9b27af"
Sep 07 22:38:49.633565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.633543    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj" event={"ID":"fd2dd1c5-0762-4da9-ab50-31edfc0f468a","Type":"ContainerDied","Data":"7bb8729956c97ed33c6ca6df26cf659befb1e24c91e5aa9d32ec6302facda004"}
Sep 07 22:38:49.633687 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.633582    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb8729956c97ed33c6ca6df26cf659befb1e24c91e5aa9d32ec6302facda004"
Sep 07 22:38:49.633735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.633684    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0b7hbj"
Sep 07 22:38:49.755050 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.755025    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:49.791903 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.791878    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gbg\" (UniqueName: \"kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg\") pod \"44049cfd-fa97-4d5e-b86a-d140c4362190\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") "
Sep 07 22:38:49.792033 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.791916    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle\") pod \"44049cfd-fa97-4d5e-b86a-d140c4362190\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") "
Sep 07 22:38:49.792033 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.791935    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util\") pod \"44049cfd-fa97-4d5e-b86a-d140c4362190\" (UID: \"44049cfd-fa97-4d5e-b86a-d140c4362190\") "
Sep 07 22:38:49.792529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.792502    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle" (OuterVolumeSpecName: "bundle") pod "44049cfd-fa97-4d5e-b86a-d140c4362190" (UID: "44049cfd-fa97-4d5e-b86a-d140c4362190"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:49.793983 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.793948    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg" (OuterVolumeSpecName: "kube-api-access-q7gbg") pod "44049cfd-fa97-4d5e-b86a-d140c4362190" (UID: "44049cfd-fa97-4d5e-b86a-d140c4362190"). InnerVolumeSpecName "kube-api-access-q7gbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:38:49.797575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.797543    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util" (OuterVolumeSpecName: "util") pod "44049cfd-fa97-4d5e-b86a-d140c4362190" (UID: "44049cfd-fa97-4d5e-b86a-d140c4362190"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:49.893669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.893598    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gbg\" (UniqueName: \"kubernetes.io/projected/44049cfd-fa97-4d5e-b86a-d140c4362190-kube-api-access-q7gbg\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:49.893669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.893629    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:49.893669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:49.893643    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44049cfd-fa97-4d5e-b86a-d140c4362190-util\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:38:50.640344 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:50.640315    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb"
Sep 07 22:38:50.640724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:50.640314    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c6zqxb" event={"ID":"44049cfd-fa97-4d5e-b86a-d140c4362190","Type":"ContainerDied","Data":"f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107"}
Sep 07 22:38:50.640724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:50.640431    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cc991509c1837858e06a74a110dc56d5f6dde7b835ca25d969055a4f2e3107"
Sep 07 22:38:52.721083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:52.721041    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:53.512544 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:53.512508    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:54.316606 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:54.316570    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:57.051946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.051911    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"]
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052262    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052274    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052287    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="util"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052293    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="util"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052301    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052306    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052313    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052318    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052325    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052329    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052334    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052339    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="pull"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052347    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="util"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052352    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="util"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052364    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052368    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="extract"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052374    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="util"
Sep 07 22:38:57.052397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052379    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="util"
Sep 07 22:38:57.052917 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052432    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b689a5fe-34ee-45fc-b1c1-c371b02d444a" containerName="extract"
Sep 07 22:38:57.052917 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052442    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="44049cfd-fa97-4d5e-b86a-d140c4362190" containerName="extract"
Sep 07 22:38:57.052917 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.052449    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2dd1c5-0762-4da9-ab50-31edfc0f468a" containerName="extract"
Sep 07 22:38:57.055621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.055604    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.058298 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.058270    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 22:38:57.058424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.058370    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 22:38:57.058703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.058684    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cczlv\""
Sep 07 22:38:57.068076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.068051    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"]
Sep 07 22:38:57.151652 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.151626    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.151762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.151671    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qk77\" (UniqueName: \"kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.252722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.252695    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.252844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.252736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qk77\" (UniqueName: \"kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.253061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.253040    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.270018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.269990    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qk77\" (UniqueName: \"kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.366738 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.366681    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:38:57.605704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.605671    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:57.659685 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.659664    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"]
Sep 07 22:38:57.662146 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:38:57.662099    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5597958e_72bc_459a_b09c_21b6e95ee46d.slice/crio-457119a7b5f0920698d93192253961bd59181d7bb9f8b08f44f51bd43214be5c WatchSource:0}: Error finding container 457119a7b5f0920698d93192253961bd59181d7bb9f8b08f44f51bd43214be5c: Status 404 returned error can't find the container with id 457119a7b5f0920698d93192253961bd59181d7bb9f8b08f44f51bd43214be5c
Sep 07 22:38:57.675434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.675408    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4" event={"ID":"5597958e-72bc-459a-b09c-21b6e95ee46d","Type":"ContainerStarted","Data":"457119a7b5f0920698d93192253961bd59181d7bb9f8b08f44f51bd43214be5c"}
Sep 07 22:38:57.754077 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:57.754044    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:59.727377 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:59.727334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:38:59.963346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:38:59.963311    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:01.455487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:01.455445    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:01.550946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:01.550913    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:02.125264 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:02.125228    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:03.551265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:03.551232    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:04.047926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.047890    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-96z4x"]
Sep 07 22:39:04.051896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.051865    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:04.054326 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.054301    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zc52n\""
Sep 07 22:39:04.061208 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.061182    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-96z4x"]
Sep 07 22:39:04.103052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.103018    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xck\" (UniqueName: \"kubernetes.io/projected/5a0c0620-609d-4df0-872a-b383bec31142-kube-api-access-h8xck\") pod \"authorino-operator-68d88948d8-96z4x\" (UID: \"5a0c0620-609d-4df0-872a-b383bec31142\") " pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:04.204110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.204079    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xck\" (UniqueName: \"kubernetes.io/projected/5a0c0620-609d-4df0-872a-b383bec31142-kube-api-access-h8xck\") pod \"authorino-operator-68d88948d8-96z4x\" (UID: \"5a0c0620-609d-4df0-872a-b383bec31142\") " pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:04.212349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.212319    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xck\" (UniqueName: \"kubernetes.io/projected/5a0c0620-609d-4df0-872a-b383bec31142-kube-api-access-h8xck\") pod \"authorino-operator-68d88948d8-96z4x\" (UID: \"5a0c0620-609d-4df0-872a-b383bec31142\") " pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:04.364912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:04.364819    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:05.027084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.027054    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-96z4x"]
Sep 07 22:39:05.028580 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:39:05.028555    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0c0620_609d_4df0_872a_b383bec31142.slice/crio-e6d75adc8b0f9c46e85cb827556d6234a15ce755acd7098252182a9154c035bc WatchSource:0}: Error finding container e6d75adc8b0f9c46e85cb827556d6234a15ce755acd7098252182a9154c035bc: Status 404 returned error can't find the container with id e6d75adc8b0f9c46e85cb827556d6234a15ce755acd7098252182a9154c035bc
Sep 07 22:39:05.466370 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.466334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:05.566175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.566115    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:05.724065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.723981    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4" event={"ID":"5597958e-72bc-459a-b09c-21b6e95ee46d","Type":"ContainerStarted","Data":"e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2"}
Sep 07 22:39:05.724251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.724093    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:39:05.725413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.725386    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x" event={"ID":"5a0c0620-609d-4df0-872a-b383bec31142","Type":"ContainerStarted","Data":"e6d75adc8b0f9c46e85cb827556d6234a15ce755acd7098252182a9154c035bc"}
Sep 07 22:39:05.743870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:05.743821    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4" podStartSLOduration=1.567616833 podStartE2EDuration="8.743807652s" totalImagesPullingTime="7.176190819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:05.742306652 +0000 UTC m=+425.584574921" watchObservedRunningTime="2026-09-07 22:39:05.743807652 +0000 UTC m=+425.586075918"
Sep 07 22:39:06.731843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:06.731813    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x" event={"ID":"5a0c0620-609d-4df0-872a-b383bec31142","Type":"ContainerStarted","Data":"39ed214b5d1625689b84d3f19585262c49d8d17982a935a5a5eba0fc31981ef4"}
Sep 07 22:39:06.732233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:06.732063    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:06.749640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:06.749589    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x" podStartSLOduration=1.145417042 podStartE2EDuration="2.749576955s" totalImagesPullingTime="1.604159913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:06.746756859 +0000 UTC m=+426.589025127" watchObservedRunningTime="2026-09-07 22:39:06.749576955 +0000 UTC m=+426.591845222"
Sep 07 22:39:07.480278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:07.480222    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:07.573152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:07.573095    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:07.922665 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:07.922587    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:07.976315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:07.976288    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:08.325211 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:08.325186    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:08.377950 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:08.377895    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:08.732422 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:08.732387    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:08.785735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:08.785708    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:14.399076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:14.399036    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:14.494836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:14.494806    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:15.883766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:15.883733    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:39:15.987860 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:15.987814    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:16.103873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:16.103844    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:16.734580 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:16.734546    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:39:17.739732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:17.739701    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-96z4x"
Sep 07 22:39:18.784348 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.784316    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"]
Sep 07 22:39:18.784844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.784559    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" containerName="manager" containerID="cri-o://e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2" gracePeriod=2
Sep 07 22:39:18.786703 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:39:18.786659    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-rhnb4\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:39:18.787655 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.787610    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"]
Sep 07 22:39:18.810299 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.810273    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:18.811023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.811000    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" containerName="manager"
Sep 07 22:39:18.811153 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.811025    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" containerName="manager"
Sep 07 22:39:18.811226 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.811170    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" containerName="manager"
Sep 07 22:39:18.814354 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.814333    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:18.830841 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.830814    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:18.844192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.844165    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"]
Sep 07 22:39:18.848307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.848287    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:18.852251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.852231    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2kqgq\""
Sep 07 22:39:18.867688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.867662    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"]
Sep 07 22:39:18.921784 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.921747    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:18.921901 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.921836    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhhk\" (UniqueName: \"kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:18.922000 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:18.921893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmrh\" (UniqueName: \"kubernetes.io/projected/3789c481-49be-4d95-8dab-83691714b20c-kube-api-access-lpmrh\") pod \"limitador-operator-controller-manager-6c549b6d85-tqxxf\" (UID: \"3789c481-49be-4d95-8dab-83691714b20c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:19.022763 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.022736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.022852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.022828    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhhk\" (UniqueName: \"kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.022852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.022850    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmrh\" (UniqueName: \"kubernetes.io/projected/3789c481-49be-4d95-8dab-83691714b20c-kube-api-access-lpmrh\") pod \"limitador-operator-controller-manager-6c549b6d85-tqxxf\" (UID: \"3789c481-49be-4d95-8dab-83691714b20c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:19.023120 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.023099    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.028243 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.028224    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:39:19.032825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.032805    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhhk\" (UniqueName: \"kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk\") pod \"kuadrant-operator-controller-manager-58fc56757c-bjczp\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.033160 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.033108    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmrh\" (UniqueName: \"kubernetes.io/projected/3789c481-49be-4d95-8dab-83691714b20c-kube-api-access-lpmrh\") pod \"limitador-operator-controller-manager-6c549b6d85-tqxxf\" (UID: \"3789c481-49be-4d95-8dab-83691714b20c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:19.124167 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.124086    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume\") pod \"5597958e-72bc-459a-b09c-21b6e95ee46d\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") "
Sep 07 22:39:19.124284 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.124187    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qk77\" (UniqueName: \"kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77\") pod \"5597958e-72bc-459a-b09c-21b6e95ee46d\" (UID: \"5597958e-72bc-459a-b09c-21b6e95ee46d\") "
Sep 07 22:39:19.124573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.124549    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5597958e-72bc-459a-b09c-21b6e95ee46d" (UID: "5597958e-72bc-459a-b09c-21b6e95ee46d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:39:19.126167 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.126148    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77" (OuterVolumeSpecName: "kube-api-access-4qk77") pod "5597958e-72bc-459a-b09c-21b6e95ee46d" (UID: "5597958e-72bc-459a-b09c-21b6e95ee46d"). InnerVolumeSpecName "kube-api-access-4qk77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:19.152015 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.151991    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:19.159920 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.159902    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.170855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.170827    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:19.226553 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.225221    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qk77\" (UniqueName: \"kubernetes.io/projected/5597958e-72bc-459a-b09c-21b6e95ee46d-kube-api-access-4qk77\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:19.226553 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.225260    2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5597958e-72bc-459a-b09c-21b6e95ee46d-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:19.257437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.257411    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:19.445101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.445069    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:19.447162 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.447138    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"]
Sep 07 22:39:19.447254 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:39:19.447122    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bc6490_666f_4565_9d37_5157185fd654.slice/crio-770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645 WatchSource:0}: Error finding container 770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645: Status 404 returned error can't find the container with id 770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645
Sep 07 22:39:19.448803 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:39:19.448781    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3789c481_49be_4d95_8dab_83691714b20c.slice/crio-bae98518f266670535ffacdaf8b3ebdc1889dbe1119a7d4e3ae69f6f820928b3 WatchSource:0}: Error finding container bae98518f266670535ffacdaf8b3ebdc1889dbe1119a7d4e3ae69f6f820928b3: Status 404 returned error can't find the container with id bae98518f266670535ffacdaf8b3ebdc1889dbe1119a7d4e3ae69f6f820928b3
Sep 07 22:39:19.803701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.803629    2550 generic.go:356] "Generic (PLEG): container finished" podID="5597958e-72bc-459a-b09c-21b6e95ee46d" containerID="e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2" exitCode=0
Sep 07 22:39:19.803701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.803684    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhnb4"
Sep 07 22:39:19.804193 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.803725    2550 scope.go:117] "RemoveContainer" containerID="e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2"
Sep 07 22:39:19.805671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.805642    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" event={"ID":"81bc6490-666f-4565-9d37-5157185fd654","Type":"ContainerStarted","Data":"3ba12de2cd9b4c43bbf9864f99b8c11010c41cc8407953c587b8dc86e8fb185e"}
Sep 07 22:39:19.805797 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.805707    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:19.805797 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.805730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" event={"ID":"81bc6490-666f-4565-9d37-5157185fd654","Type":"ContainerStarted","Data":"770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645"}
Sep 07 22:39:19.807352 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.807328    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf" event={"ID":"3789c481-49be-4d95-8dab-83691714b20c","Type":"ContainerStarted","Data":"bae98518f266670535ffacdaf8b3ebdc1889dbe1119a7d4e3ae69f6f820928b3"}
Sep 07 22:39:19.819439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.819422    2550 scope.go:117] "RemoveContainer" containerID="e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2"
Sep 07 22:39:19.819679 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:39:19.819663    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2\": container with ID starting with e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2 not found: ID does not exist" containerID="e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2"
Sep 07 22:39:19.819745 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.819685    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2"} err="failed to get container status \"e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2\": rpc error: code = NotFound desc = could not find container \"e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2\": container with ID starting with e7aa32f6319b2ae8813107f667e12fa76e059055ad3ed840551c67c1a2e507a2 not found: ID does not exist"
Sep 07 22:39:19.828744 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:19.828701    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" podStartSLOduration=1.8286852059999998 podStartE2EDuration="1.828685206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:19.826188936 +0000 UTC m=+439.668457204" watchObservedRunningTime="2026-09-07 22:39:19.828685206 +0000 UTC m=+439.670953473"
Sep 07 22:39:20.345095 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.345061    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:20.446410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.446297    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:20.796371 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.796344    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5597958e-72bc-459a-b09c-21b6e95ee46d" path="/var/lib/kubelet/pods/5597958e-72bc-459a-b09c-21b6e95ee46d/volumes"
Sep 07 22:39:20.814771 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.814739    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf" event={"ID":"3789c481-49be-4d95-8dab-83691714b20c","Type":"ContainerStarted","Data":"98c74b43626a9d8ea345c46be00d411b1b9c08b04c5ad18a771779f0fec24967"}
Sep 07 22:39:20.815053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.814891    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:20.830658 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:20.830614    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf" podStartSLOduration=1.536092732 podStartE2EDuration="2.830600745s" totalImagesPullingTime="1.294508013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:20.828971017 +0000 UTC m=+440.671239285" watchObservedRunningTime="2026-09-07 22:39:20.830600745 +0000 UTC m=+440.672869012"
Sep 07 22:39:23.934074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:23.933988    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:24.034069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:24.034021    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.135389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.135321    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.185325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.185297    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.541943 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.541906    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.595502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.595461    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.748871 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.748831    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:29.849912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:29.849807    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:30.816874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:30.816842    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:31.821878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:31.821850    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-tqxxf"
Sep 07 22:39:34.658933 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.658888    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:34.659464 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.659162    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" podUID="81bc6490-666f-4565-9d37-5157185fd654" containerName="manager" containerID="cri-o://3ba12de2cd9b4c43bbf9864f99b8c11010c41cc8407953c587b8dc86e8fb185e" gracePeriod=10
Sep 07 22:39:34.891445 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.891414    2550 generic.go:356] "Generic (PLEG): container finished" podID="81bc6490-666f-4565-9d37-5157185fd654" containerID="3ba12de2cd9b4c43bbf9864f99b8c11010c41cc8407953c587b8dc86e8fb185e" exitCode=0
Sep 07 22:39:34.891575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.891474    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" event={"ID":"81bc6490-666f-4565-9d37-5157185fd654","Type":"ContainerDied","Data":"3ba12de2cd9b4c43bbf9864f99b8c11010c41cc8407953c587b8dc86e8fb185e"}
Sep 07 22:39:34.891575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.891528    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp" event={"ID":"81bc6490-666f-4565-9d37-5157185fd654","Type":"ContainerDied","Data":"770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645"}
Sep 07 22:39:34.891575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.891547    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770b7bfd1316f523a287e918e50692b9a3cccbd1c42ae52864e93bd69a69f645"
Sep 07 22:39:34.905045 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:34.905020    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:35.058201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.058172    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume\") pod \"81bc6490-666f-4565-9d37-5157185fd654\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") "
Sep 07 22:39:35.058365 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.058263    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkhhk\" (UniqueName: \"kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk\") pod \"81bc6490-666f-4565-9d37-5157185fd654\" (UID: \"81bc6490-666f-4565-9d37-5157185fd654\") "
Sep 07 22:39:35.058537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.058510    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "81bc6490-666f-4565-9d37-5157185fd654" (UID: "81bc6490-666f-4565-9d37-5157185fd654"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:39:35.060176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.060152    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk" (OuterVolumeSpecName: "kube-api-access-qkhhk") pod "81bc6490-666f-4565-9d37-5157185fd654" (UID: "81bc6490-666f-4565-9d37-5157185fd654"). InnerVolumeSpecName "kube-api-access-qkhhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:35.159518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.159493    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkhhk\" (UniqueName: \"kubernetes.io/projected/81bc6490-666f-4565-9d37-5157185fd654-kube-api-access-qkhhk\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:35.159518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.159514    2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81bc6490-666f-4565-9d37-5157185fd654-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:35.896244 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.896213    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"
Sep 07 22:39:35.917341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.917311    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:35.920793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:35.920766    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bjczp"]
Sep 07 22:39:36.220946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:36.220916    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:36.319235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:36.319208    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:36.796119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:36.796087    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81bc6490-666f-4565-9d37-5157185fd654" path="/var/lib/kubelet/pods/81bc6490-666f-4565-9d37-5157185fd654/volumes"
Sep 07 22:39:40.904108 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:40.904069    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f7f687dff-nqq6l" podUID="97557405-ef80-46f4-92c7-972abcc3a353" containerName="console" containerID="cri-o://09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a" gracePeriod=15
Sep 07 22:39:40.949284 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:40.949248    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:41.050523 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.050489    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:41.151789 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.151766    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:39:41.301139 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301104    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301188    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301281    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301319    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301503 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301346    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301503 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301374    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxw6w\" (UniqueName: \"kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301503 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301401    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert\") pod \"97557405-ef80-46f4-92c7-972abcc3a353\" (UID: \"97557405-ef80-46f4-92c7-972abcc3a353\") "
Sep 07 22:39:41.301681 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301653    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca" (OuterVolumeSpecName: "service-ca") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:39:41.301681 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301663    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:39:41.301794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301673    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:39:41.301794 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.301699    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config" (OuterVolumeSpecName: "console-config") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:39:41.303341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.303317    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:39:41.303540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.303525    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:39:41.303682 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.303663    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w" (OuterVolumeSpecName: "kube-api-access-nxw6w") pod "97557405-ef80-46f4-92c7-972abcc3a353" (UID: "97557405-ef80-46f4-92c7-972abcc3a353"). InnerVolumeSpecName "kube-api-access-nxw6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:41.402269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402246    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402266    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-oauth-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402275    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-console-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402284    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxw6w\" (UniqueName: \"kubernetes.io/projected/97557405-ef80-46f4-92c7-972abcc3a353-kube-api-access-nxw6w\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402293    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402303    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97557405-ef80-46f4-92c7-972abcc3a353-console-oauth-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.402416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.402311    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97557405-ef80-46f4-92c7-972abcc3a353-trusted-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:39:41.928629 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.928596    2550 generic.go:356] "Generic (PLEG): container finished" podID="97557405-ef80-46f4-92c7-972abcc3a353" containerID="09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a" exitCode=0
Sep 07 22:39:41.929032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.928639    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f687dff-nqq6l"
Sep 07 22:39:41.929032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.928680    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f687dff-nqq6l" event={"ID":"97557405-ef80-46f4-92c7-972abcc3a353","Type":"ContainerDied","Data":"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"}
Sep 07 22:39:41.929032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.928731    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f687dff-nqq6l" event={"ID":"97557405-ef80-46f4-92c7-972abcc3a353","Type":"ContainerDied","Data":"3a857e10467b031fd5e0368469a715e0b021328fa89c9d4ef4ae7dfa87b01259"}
Sep 07 22:39:41.929032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.928755    2550 scope.go:117] "RemoveContainer" containerID="09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"
Sep 07 22:39:41.938086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.938069    2550 scope.go:117] "RemoveContainer" containerID="09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"
Sep 07 22:39:41.938370 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:39:41.938354    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a\": container with ID starting with 09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a not found: ID does not exist" containerID="09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"
Sep 07 22:39:41.938416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.938376    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a"} err="failed to get container status \"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a\": rpc error: code = NotFound desc = could not find container \"09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a\": container with ID starting with 09fc4565cd85cb5dc64d6b70fcbc736247327650ead0f5f4775fc1600a5c268a not found: ID does not exist"
Sep 07 22:39:41.949951 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.949906    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:39:41.951650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:41.951631    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7f687dff-nqq6l"]
Sep 07 22:39:42.796296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:42.796263    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97557405-ef80-46f4-92c7-972abcc3a353" path="/var/lib/kubelet/pods/97557405-ef80-46f4-92c7-972abcc3a353/volumes"
Sep 07 22:39:45.747033 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:45.746991    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:45.846207 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:45.846174    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:50.942833 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:50.942797    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:39:51.233705 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:39:51.233631    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:07.657121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.657083    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:07.764333 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.764297    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:07.811959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.811932    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:07.960005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.959969    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:40:07.960492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960422    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81bc6490-666f-4565-9d37-5157185fd654" containerName="manager"
Sep 07 22:40:07.960492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960448    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="81bc6490-666f-4565-9d37-5157185fd654" containerName="manager"
Sep 07 22:40:07.960492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960466    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97557405-ef80-46f4-92c7-972abcc3a353" containerName="console"
Sep 07 22:40:07.960492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960476    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="97557405-ef80-46f4-92c7-972abcc3a353" containerName="console"
Sep 07 22:40:07.960775 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960592    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="97557405-ef80-46f4-92c7-972abcc3a353" containerName="console"
Sep 07 22:40:07.960775 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.960610    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="81bc6490-666f-4565-9d37-5157185fd654" containerName="manager"
Sep 07 22:40:07.964760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.964745    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:07.966977 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.966958    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 22:40:07.967106 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.966998    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nwtjv\""
Sep 07 22:40:07.972235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.972010    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:40:07.997155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:07.997117    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:40:08.100608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.100576    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn4h\" (UniqueName: \"kubernetes.io/projected/59b03edd-f91a-4d62-af25-1816da1d8641-kube-api-access-lhn4h\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.100731 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.100666    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59b03edd-f91a-4d62-af25-1816da1d8641-config-file\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.201158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.201123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn4h\" (UniqueName: \"kubernetes.io/projected/59b03edd-f91a-4d62-af25-1816da1d8641-kube-api-access-lhn4h\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.201266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.201192    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59b03edd-f91a-4d62-af25-1816da1d8641-config-file\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.201720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.201703    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/59b03edd-f91a-4d62-af25-1816da1d8641-config-file\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.208254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.208227    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn4h\" (UniqueName: \"kubernetes.io/projected/59b03edd-f91a-4d62-af25-1816da1d8641-kube-api-access-lhn4h\") pod \"limitador-limitador-85d9db4d5d-lv56r\" (UID: \"59b03edd-f91a-4d62-af25-1816da1d8641\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.277123 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.277067    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:08.470164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.470122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:08.551110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.551015    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:40:08.552816 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:40:08.552791    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b03edd_f91a_4d62_af25_1816da1d8641.slice/crio-a980fcc41add2d268cc2c65446753182de55abceaf0f010276c69320f294b0a9 WatchSource:0}: Error finding container a980fcc41add2d268cc2c65446753182de55abceaf0f010276c69320f294b0a9: Status 404 returned error can't find the container with id a980fcc41add2d268cc2c65446753182de55abceaf0f010276c69320f294b0a9
Sep 07 22:40:08.553515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:08.553487    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:09.070420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:09.070386    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r" event={"ID":"59b03edd-f91a-4d62-af25-1816da1d8641","Type":"ContainerStarted","Data":"a980fcc41add2d268cc2c65446753182de55abceaf0f010276c69320f294b0a9"}
Sep 07 22:40:10.077394 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:10.077354    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r" event={"ID":"59b03edd-f91a-4d62-af25-1816da1d8641","Type":"ContainerStarted","Data":"5c83ce311054f16519205e642af0c9db57b6db864de930df8dd9b3ecaa8a9cfd"}
Sep 07 22:40:10.077775 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:10.077532    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:10.093204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:10.093157    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r" podStartSLOduration=2.032776981 podStartE2EDuration="3.093142372s" totalImagesPullingTime="1.060365391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:10.090705382 +0000 UTC m=+489.932973649" watchObservedRunningTime="2026-09-07 22:40:10.093142372 +0000 UTC m=+489.935410635"
Sep 07 22:40:21.083856 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:21.083822    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"
Sep 07 22:40:24.406385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:24.406350    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:24.506979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:24.506947    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:24.863157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:24.863105    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:37.255669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:37.255632    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:37.813413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:37.813370    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:43.542365 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:43.542321    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:43.641995 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:43.641961    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:43.719027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:43.718992    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:43.791112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:43.791086    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:47.203202 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:47.203148    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:40:47.300252 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:40:47.300213    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:13.038155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:13.038044    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:13.092001 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:13.091970    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:18.807174 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:18.807122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:18.906799 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:18.906750    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.219727 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.219685    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.327375 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.327337    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.642166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.642067    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.743862 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.743824    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.842772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.842741    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:20.927992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:20.927921    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:21.028948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:21.028916    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:41:31.161359 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:31.161323    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:31.260341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:31.260309    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:49.380948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:49.380915    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:41:49.778156 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:41:49.778111    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:42:00.390206 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:00.390167    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:01.066859 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:01.066813    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:42:10.966485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.966451    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"]
Sep 07 22:42:10.969872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.969855    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:10.972109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.972085    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tklfr\""
Sep 07 22:42:10.972315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.972290    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 22:42:10.973199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.973178    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 22:42:10.973309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.973266    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 07 22:42:10.977924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:10.977905    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"]
Sep 07 22:42:11.001520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001493    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.001610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001529    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.001610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001560    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.001610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001602    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.001740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001622    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7g4\" (UniqueName: \"kubernetes.io/projected/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-kube-api-access-gn7g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.001740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.001710    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.069605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.069569    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:11.102944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.102919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.102951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.102980    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103024    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103059    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103084    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7g4\" (UniqueName: \"kubernetes.io/projected/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-kube-api-access-gn7g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103377 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103407    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.103461 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.103437    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.105268 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.105239    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.105485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.105469    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.110325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.110300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7g4\" (UniqueName: \"kubernetes.io/projected/f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44-kube-api-access-gn7g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h\" (UID: \"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.282397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.282317    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:11.558101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.558073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"]
Sep 07 22:42:11.560036 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:42:11.560000    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08fdd25_c8bc_4b51_92e1_f2f2c3c8ec44.slice/crio-4fee0da5336ab29bdaf5ed64a11c4e4c1d39abc6367fbbb0c5cb8815011b945e WatchSource:0}: Error finding container 4fee0da5336ab29bdaf5ed64a11c4e4c1d39abc6367fbbb0c5cb8815011b945e: Status 404 returned error can't find the container with id 4fee0da5336ab29bdaf5ed64a11c4e4c1d39abc6367fbbb0c5cb8815011b945e
Sep 07 22:42:11.561334 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.561316    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:42:11.699779 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:11.699746    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h" event={"ID":"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44","Type":"ContainerStarted","Data":"4fee0da5336ab29bdaf5ed64a11c4e4c1d39abc6367fbbb0c5cb8815011b945e"}
Sep 07 22:42:12.072368 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:12.072320    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:42:12.706304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:12.706275    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h" event={"ID":"f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44","Type":"ContainerStarted","Data":"a9890f31d7892f0033661fbfbef997348fef5a5d6e96f0130ba5cb26165d7716"}
Sep 07 22:42:12.723548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:12.723503    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h" podStartSLOduration=1.75213754 podStartE2EDuration="2.723487036s" totalImagesPullingTime="971.349496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:42:12.722213848 +0000 UTC m=+612.564482116" watchObservedRunningTime="2026-09-07 22:42:12.723487036 +0000 UTC m=+612.565755303"
Sep 07 22:42:19.864056 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.864018    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"]
Sep 07 22:42:19.868053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.868029    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.870430 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.870405    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 07 22:42:19.877304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.877280    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"]
Sep 07 22:42:19.889830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.889808    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:19.966342 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.966595 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966574    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrxt\" (UniqueName: \"kubernetes.io/projected/87dee3d6-bed9-4e64-b22a-35c607427c3f-kube-api-access-fxrxt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.966741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.966886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.967003 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966918    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:19.967003 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:19.966937    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dee3d6-bed9-4e64-b22a-35c607427c3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067232    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067317    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067344    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dee3d6-bed9-4e64-b22a-35c607427c3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067464    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrxt\" (UniqueName: \"kubernetes.io/projected/87dee3d6-bed9-4e64-b22a-35c607427c3f-kube-api-access-fxrxt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067721 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067701    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067781 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067742    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.067864 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.067839    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.069646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.069629    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87dee3d6-bed9-4e64-b22a-35c607427c3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.070023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.070004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dee3d6-bed9-4e64-b22a-35c607427c3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.074271 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.074250    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrxt\" (UniqueName: \"kubernetes.io/projected/87dee3d6-bed9-4e64-b22a-35c607427c3f-kube-api-access-fxrxt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dbptf\" (UID: \"87dee3d6-bed9-4e64-b22a-35c607427c3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.178947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.178893    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:20.449015 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.448987    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"]
Sep 07 22:42:20.450376 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:42:20.450348    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dee3d6_bed9_4e64_b22a_35c607427c3f.slice/crio-fa2bbe97f259c1f1dbe6db7d533a0ff67d7234f2f2f7dbc2664f5a1a6f72f80d WatchSource:0}: Error finding container fa2bbe97f259c1f1dbe6db7d533a0ff67d7234f2f2f7dbc2664f5a1a6f72f80d: Status 404 returned error can't find the container with id fa2bbe97f259c1f1dbe6db7d533a0ff67d7234f2f2f7dbc2664f5a1a6f72f80d
Sep 07 22:42:20.749163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.749068    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf" event={"ID":"87dee3d6-bed9-4e64-b22a-35c607427c3f","Type":"ContainerStarted","Data":"a138be1446e2350d0d4fad9134786bc727304a95a0afc66cb4826071d075eaa9"}
Sep 07 22:42:20.749163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.749155    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf" event={"ID":"87dee3d6-bed9-4e64-b22a-35c607427c3f","Type":"ContainerStarted","Data":"fa2bbe97f259c1f1dbe6db7d533a0ff67d7234f2f2f7dbc2664f5a1a6f72f80d"}
Sep 07 22:42:20.764994 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:20.764948    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf" podStartSLOduration=1.584021017 podStartE2EDuration="1.764933283s" totalImagesPullingTime="180.912266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:42:20.763728654 +0000 UTC m=+620.605996923" watchObservedRunningTime="2026-09-07 22:42:20.764933283 +0000 UTC m=+620.607201571"
Sep 07 22:42:21.263642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:21.263606    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:42:21.282980 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:21.282950    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:21.283147 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:21.283004    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:21.295368 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:21.295346    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:21.765897 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:21.765865    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h"
Sep 07 22:42:27.798957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:27.798921    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:29.765195 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:29.765160    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:42:30.179109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:30.179035    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:30.179109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:30.179091    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:30.191509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:30.191482    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:30.814877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:30.814843    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dbptf"
Sep 07 22:42:34.549568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:34.549534    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:36.084706 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:36.084670    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:52.536258 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:52.536223    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:52.586862 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:52.586830    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:42:58.397522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:42:58.397486    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:43:00.268494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:43:00.268449    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:43:24.506061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:43:24.506029    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:43:26.572484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:43:26.572450    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:44:10.728360 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:10.728268    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:44:10.777587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:10.777554    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:44:45.734969 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:45.734914    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:44:45.821745 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:45.821711    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:44:46.808189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:46.808146    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:44:46.906025 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:44:46.905995    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:05.990019 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:05.989989    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:06.099240 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:06.099212    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:06.308847 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:06.308739    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:06.810947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:06.810910    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:07.323114 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:07.323080    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:07.429240 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:07.429206    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:07.740406 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:07.740378    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:09.461782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:09.461743    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:09.556550 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:09.556513    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:09.777147 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:09.777056    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:45:10.915896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:10.915863    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:11.016623 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:11.016586    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:11.336846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:11.336794    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:11.427103 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:11.427069    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:11.494674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:11.494629    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:13.947248 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:13.947204    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.046880 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.046834    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.165568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.165529    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.283398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.283315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.432028 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.431993    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:45:14.435921 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.435897    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.440278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.440252    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e3b5bcb-openshift-default-dockercfg-txqsl\""
Sep 07 22:45:14.456838 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.456813    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:45:14.492789 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.492749    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.536722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536654    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536697    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536755    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536820    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536848    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536879    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536940    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.536984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.536977    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz57g\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637591 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637561    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637596    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637617    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637641    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz57g\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637760    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637827    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.637892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.637857    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.640229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.638552    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.640229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.638818    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.640229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.638830    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.640229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.639220    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.649735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.640719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.649735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.641004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.649735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.648252    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.653738 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.653716    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.653877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.653855    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz57g\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g\") pod \"e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") " pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.747584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.747553    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:14.833257 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.833226    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:14.934222 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:14.934185    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:15.031050 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:15.031019    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:45:15.033521 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:45:15.033495    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26394a83_25c2_4a07_8ac6_e80ce1b36946.slice/crio-88ff097de549a0ef52ed09a4c1d7322685794c12388a0031d5244f33e79ce981 WatchSource:0}: Error finding container 88ff097de549a0ef52ed09a4c1d7322685794c12388a0031d5244f33e79ce981: Status 404 returned error can't find the container with id 88ff097de549a0ef52ed09a4c1d7322685794c12388a0031d5244f33e79ce981
Sep 07 22:45:15.664658 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:15.664623    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerStarted","Data":"88ff097de549a0ef52ed09a4c1d7322685794c12388a0031d5244f33e79ce981"}
Sep 07 22:45:17.584638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:17.584581    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:17.684105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:17.684054    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:18.165095 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.165057    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:18.165235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.165149    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:18.165235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.165179    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:18.684066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.684036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerStarted","Data":"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"}
Sep 07 22:45:18.704870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.704825    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podStartSLOduration=1.5755373430000001 podStartE2EDuration="4.704810167s" totalImagesPullingTime="3.129272824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:18.702068664 +0000 UTC m=+798.544336931" watchObservedRunningTime="2026-09-07 22:45:18.704810167 +0000 UTC m=+798.547078434"
Sep 07 22:45:18.748197 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.748172    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:18.831680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.831632    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:18.885806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:18.885780    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:19.064979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:19.064952    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:19.292189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:19.292156    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:19.360716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:19.360641    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:19.752867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:19.752834    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:20.135415 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:20.135335    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:20.225515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:20.225484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:20.699291 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:20.699265    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:20.700117 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:20.700096    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:21.463767 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:21.463710    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:21.551351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:21.551317    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:21.650884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:21.650850    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:22.028394 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:22.028367    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:22.122252 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:22.122218    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:24.849478 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:24.849397    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:25.001456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:25.001420    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:25.097693 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:25.097653    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:25.198569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:25.198534    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:25.576993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:25.576915    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:45:27.886843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:27.886799    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:29.377369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:29.377329    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:30.968048 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:30.968002    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:31.085737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:31.085703    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:31.182434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:31.182401    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:32.832024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:32.831988    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:34.347315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.347281    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:34.653329 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.653245    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:34.909820 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.909723    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:45:34.914165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.914139    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:34.916935 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.916910    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-79b9aa-openshift-default-dockercfg-6rtxb\""
Sep 07 22:45:34.925923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.925898    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:45:34.943358 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:34.943306    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:35.017882 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.017842    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.017920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.017954    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.017984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.018007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctd2j\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.018024    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.018042    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.018076    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.018484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.018115    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.118704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118668    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.118895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.118895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctd2j\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.118895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118850    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.118895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118890    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.118974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119032    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119113    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119121    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119372    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119521 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119467    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119643    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.119738 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.119657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.121950 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.121922    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.122277 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.122257    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.126447 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.126420    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.126542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.126462    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctd2j\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j\") pod \"e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") " pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.166618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.166544    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:35.229541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.229515    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:35.535378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.521662    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:45:35.535378 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:45:35.524306    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e11d1a1_c014_45c1_8e8a_cac7ff8b2ee8.slice/crio-c1210b9a95e0a312a80ef9f72a6115e2f74fe8cd0e8b38157ccbb350b8e13327 WatchSource:0}: Error finding container c1210b9a95e0a312a80ef9f72a6115e2f74fe8cd0e8b38157ccbb350b8e13327: Status 404 returned error can't find the container with id c1210b9a95e0a312a80ef9f72a6115e2f74fe8cd0e8b38157ccbb350b8e13327
Sep 07 22:45:35.535378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.526873    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.535378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.526949    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.535378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.526993    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.700843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.700806    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:35.700999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.700859    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:35.791842 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.791730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerStarted","Data":"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"}
Sep 07 22:45:35.791842 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.791799    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerStarted","Data":"c1210b9a95e0a312a80ef9f72a6115e2f74fe8cd0e8b38157ccbb350b8e13327"}
Sep 07 22:45:35.793654 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.793626    2550 generic.go:356] "Generic (PLEG): container finished" podID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerID="9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5" exitCode=0
Sep 07 22:45:35.793791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.793698    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerDied","Data":"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"}
Sep 07 22:45:35.794060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.794045    2550 scope.go:117] "RemoveContainer" containerID="9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"
Sep 07 22:45:35.795257 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.795229    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.795357 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.795310    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.795416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.795354    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:35.814024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:35.813950    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podStartSLOduration=1.81393198 podStartE2EDuration="1.81393198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:35.810677701 +0000 UTC m=+815.652945968" watchObservedRunningTime="2026-09-07 22:45:35.81393198 +0000 UTC m=+815.656200250"
Sep 07 22:45:36.229703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.229655    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:36.234783 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.234758    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:36.800420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.800379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerStarted","Data":"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"}
Sep 07 22:45:36.800800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.800722    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:36.801432 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.801412    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:36.862569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.862508    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:36.969062 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:36.968976    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:37.062323 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.062187    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:37.191558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.191528    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:37.724399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.724365    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:37.747992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.747955    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:37.748196 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.748121    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:37.748278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.748196    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:37.826832 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.826799    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:37.893280 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:37.893247    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:38.748685 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:38.748647    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:38.748964 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:38.748936    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:38.944090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:38.944064    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:39.045039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.044944    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:39.145577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.145537    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:39.435494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.435403    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:39.504761 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.504731    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:39.748294 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.748252    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:39.748494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.748335    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:39.993584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:39.993549    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:40.748180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:40.748145    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:40.748449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:40.748413    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:40.809181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:40.809152    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:41.499707 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:41.499671    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:41.748855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:41.748809    2550 patch_prober.go:28] interesting pod/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 07 22:45:41.749066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:41.748892    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 07 22:45:42.393296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:42.393262    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:42.493527 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:42.493491    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:42.751997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:42.751970    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:42.752388 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:42.752298    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:42.753047 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:42.753030    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:45:43.250147 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:43.250098    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:43.348949 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:43.348918    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:45.567311 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.567277    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"]
Sep 07 22:45:45.575082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.575055    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:45.576201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.576175    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"]
Sep 07 22:45:45.577424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.577402    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:45:45.577529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.577421    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:45:45.623000 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.622968    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476\" (UID: \"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:45.724098 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.724062    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476\" (UID: \"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:45.724705 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.724684    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476\" (UID: \"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:45.887639 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:45.887557    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:46.156978 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:46.156941    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"]
Sep 07 22:45:46.158315 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:45:46.158291    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab91a75_b4a7_4f26_978a_6f5f7a9afc5f.slice/crio-3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9 WatchSource:0}: Error finding container 3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9: Status 404 returned error can't find the container with id 3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9
Sep 07 22:45:46.854593 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:46.854552    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476" event={"ID":"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f","Type":"ContainerStarted","Data":"3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9"}
Sep 07 22:45:46.883499 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:46.883468    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:45:48.870976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:48.870938    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476" event={"ID":"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f","Type":"ContainerStarted","Data":"340497811cb7cd96a32c26adeb806e15a94fcec4f59214cd96b20306d8daa21e"}
Sep 07 22:45:48.885514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:48.885466    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476" podStartSLOduration=1.581142402 podStartE2EDuration="3.885446769s" totalImagesPullingTime="2.304304367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:48.883786989 +0000 UTC m=+828.726055269" watchObservedRunningTime="2026-09-07 22:45:48.885446769 +0000 UTC m=+828.727715036"
Sep 07 22:45:49.878275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:49.878179    2550 generic.go:356] "Generic (PLEG): container finished" podID="3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" containerID="340497811cb7cd96a32c26adeb806e15a94fcec4f59214cd96b20306d8daa21e" exitCode=0
Sep 07 22:45:49.878275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:49.878254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476" event={"ID":"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f","Type":"ContainerDied","Data":"340497811cb7cd96a32c26adeb806e15a94fcec4f59214cd96b20306d8daa21e"}
Sep 07 22:45:51.037974 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.037950    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:51.084975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.084947    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca\") pod \"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f\" (UID: \"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f\") "
Sep 07 22:45:51.085300 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.085276    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" (UID: "3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:45:51.186095 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.186004    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:45:51.891280 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.891240    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"
Sep 07 22:45:51.891280 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.891263    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476" event={"ID":"3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f","Type":"ContainerDied","Data":"3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9"}
Sep 07 22:45:51.891598 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:51.891320    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2415c83ff01985301bebddbb8483c69036256b625339c0124a2232db152bd9"
Sep 07 22:45:56.919064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.918978    2550 generic.go:356] "Generic (PLEG): container finished" podID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerID="65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4" exitCode=0
Sep 07 22:45:56.919597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.919054    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerDied","Data":"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"}
Sep 07 22:45:56.919597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.919508    2550 scope.go:117] "RemoveContainer" containerID="65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"
Sep 07 22:45:56.920608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.920581    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:56.920696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.920647    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:56.920696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:56.920679    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:45:57.926883 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:57.926848    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerStarted","Data":"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"}
Sep 07 22:45:58.230434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.230395    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:45:58.230642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.230618    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 22:45:58.230724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.230674    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 22:45:58.755338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.755300    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:45:58.759178 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.756118    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" containerName="revoke-keys"
Sep 07 22:45:58.759350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.759338    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" containerName="revoke-keys"
Sep 07 22:45:58.759576 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.759563    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" containerName="revoke-keys"
Sep 07 22:45:58.777648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.777615    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:58.779395 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.779365    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:45:58.779495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.779458    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.782087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.782064    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-08f59294-kserve-self-signed-certs\""
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869667    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869752    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8492\" (UniqueName: \"kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869801    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.870152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.869837    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.970923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.970893    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.970923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.970928    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.970957    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8492\" (UniqueName: \"kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.970983    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.971013    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.971037    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.971458    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.971520    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.971640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.971533    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.973430 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.973408    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.973648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.973630    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:58.978032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:58.978012    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8492\" (UniqueName: \"kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492\") pod \"e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") " pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:59.089109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.089012    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:45:59.229961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.229926    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 22:45:59.230163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.229988    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 22:45:59.361564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.359832    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:45:59.878678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.878641    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:45:59.939089 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.939052    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" event={"ID":"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab","Type":"ContainerStarted","Data":"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"}
Sep 07 22:45:59.939295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:45:59.939094    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" event={"ID":"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab","Type":"ContainerStarted","Data":"fe43c830a0c934156250e8ab1e7b82ca687634917f1237a6cedcae394d1dce0e"}
Sep 07 22:46:00.230617 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:00.230583    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 22:46:00.231002 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:00.230632    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 22:46:00.649537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:00.649439    2550 scope.go:117] "RemoveContainer" containerID="3ba12de2cd9b4c43bbf9864f99b8c11010c41cc8407953c587b8dc86e8fb185e"
Sep 07 22:46:01.230219 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:01.230177    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 22:46:01.230402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:01.230236    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 22:46:01.488013 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:01.487866    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.024444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.024399    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.092504 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.092471    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.159150 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.158750    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.207791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.207713    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" podStartSLOduration=4.207695885 podStartE2EDuration="4.207695885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:59.966257289 +0000 UTC m=+839.808525560" watchObservedRunningTime="2026-09-07 22:46:02.207695885 +0000 UTC m=+842.049964145"
Sep 07 22:46:02.208957 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.208928    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"]
Sep 07 22:46:02.210988 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.210953    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3b5bcb-h4476"]
Sep 07 22:46:02.215251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.215231    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.229898 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.229870    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 22:46:02.229986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.229914    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 22:46:02.330176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.330083    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:46:02.330366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.330316    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy" containerID="cri-o://0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba" gracePeriod=30
Sep 07 22:46:02.360509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.360313    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.667708 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.667613    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.678039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.678010    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:46:02.797120 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.797077    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f" path="/var/lib/kubelet/pods/3ab91a75-b4a7-4f26-978a-6f5f7a9afc5f/volumes"
Sep 07 22:46:02.802476 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.802446    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:02.899618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:02.899568    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:03.059014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.058972    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:03.300764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.300725    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:03.306648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.306625    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.309164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.309093    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a602f13b-gw-openshift-default-dockercfg-d5s5t\""
Sep 07 22:46:03.330823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.330790    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:03.333827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.333803    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:03.413362 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413322    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413362 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413362    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413456    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pdw\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413635    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413677    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.413831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.413725    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.420069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.420031    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:46:03.420579 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.420548    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:46:03.421855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.421831    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:46:03.480621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.480582    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:03.514844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.514809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515006 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.514862    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515154    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pdw\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515234    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515343 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515324    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515400    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515483 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515483 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515477    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515600 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515547    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515657 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515598    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515750 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515731    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.515945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.515917    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.516232 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.516196    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.516383 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.516270    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.517656 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.517631    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.517778 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.517714    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.522837 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.522813    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pdw\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.523160 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.523140    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token\") pod \"e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") " pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.633001 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.632927    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:03.923562 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.923535    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:03.925535 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:46:03.925509    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c9955f_82ff_4290_a464_e5d605af995f.slice/crio-d168404c8d422bb93d39f92fc70f7537ac005ed169fc89dcd80c4d8c8b9a937b WatchSource:0}: Error finding container d168404c8d422bb93d39f92fc70f7537ac005ed169fc89dcd80c4d8c8b9a937b: Status 404 returned error can't find the container with id d168404c8d422bb93d39f92fc70f7537ac005ed169fc89dcd80c4d8c8b9a937b
Sep 07 22:46:03.927884 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.927855    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:03.927942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.927918    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:03.927975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.927945    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:03.970506 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:03.970477    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerStarted","Data":"d168404c8d422bb93d39f92fc70f7537ac005ed169fc89dcd80c4d8c8b9a937b"}
Sep 07 22:46:04.977327 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:04.977293    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerStarted","Data":"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"}
Sep 07 22:46:04.995621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:04.995570    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podStartSLOduration=1.995554028 podStartE2EDuration="1.995554028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:04.994260456 +0000 UTC m=+844.836528723" watchObservedRunningTime="2026-09-07 22:46:04.995554028 +0000 UTC m=+844.837822296"
Sep 07 22:46:05.633053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:05.633020    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:05.638065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:05.638044    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:05.982984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:05.982956    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:05.983935 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:05.983917    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:06.437808 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:06.437703    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:06.537199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:06.537118    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:06.722996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:06.722963    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:06.823099 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:06.823060    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:07.107565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.107460    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:07.208069 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.208040    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:07.282505 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.282464    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:07.586717 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.586694    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:46:07.655196 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655167    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655314 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655216    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655314 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655246    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655360    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655418    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655462    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data" (OuterVolumeSpecName: "istio-data") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:07.655548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655498    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655555    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz57g\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655619    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655645    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket\") pod \"26394a83-25c2-4a07-8ac6-e80ce1b36946\" (UID: \"26394a83-25c2-4a07-8ac6-e80ce1b36946\") "
Sep 07 22:46:07.655746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655723    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:07.655857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655817    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:07.655968 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655919    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:07.656029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.655967    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:07.656029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.656019    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.656106 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.656038    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26394a83-25c2-4a07-8ac6-e80ce1b36946-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.656106 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.656056    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.657473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.657451    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:46:07.657607 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.657584    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g" (OuterVolumeSpecName: "kube-api-access-vz57g") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "kube-api-access-vz57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:07.657730 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.657716    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token" (OuterVolumeSpecName: "istio-token") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:07.657824 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.657806    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26394a83-25c2-4a07-8ac6-e80ce1b36946" (UID: "26394a83-25c2-4a07-8ac6-e80ce1b36946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757268    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz57g\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-kube-api-access-vz57g\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757289    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757300    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757309    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757317    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:07.757321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:07.757324    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26394a83-25c2-4a07-8ac6-e80ce1b36946-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:08.001318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.001280    2550 generic.go:356] "Generic (PLEG): container finished" podID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerID="0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba" exitCode=0
Sep 07 22:46:08.001481 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.001340    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"
Sep 07 22:46:08.001481 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.001359    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerDied","Data":"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"}
Sep 07 22:46:08.001481 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.001432    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn" event={"ID":"26394a83-25c2-4a07-8ac6-e80ce1b36946","Type":"ContainerDied","Data":"88ff097de549a0ef52ed09a4c1d7322685794c12388a0031d5244f33e79ce981"}
Sep 07 22:46:08.001481 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.001452    2550 scope.go:117] "RemoveContainer" containerID="0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"
Sep 07 22:46:08.010514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.010498    2550 scope.go:117] "RemoveContainer" containerID="9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"
Sep 07 22:46:08.018212 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.018187    2550 scope.go:117] "RemoveContainer" containerID="0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"
Sep 07 22:46:08.018486 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:46:08.018468    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba\": container with ID starting with 0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba not found: ID does not exist" containerID="0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"
Sep 07 22:46:08.018540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.018494    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba"} err="failed to get container status \"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba\": rpc error: code = NotFound desc = could not find container \"0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba\": container with ID starting with 0f4f3a4a8156088be03044437b84a246e8d343cc5f44ade068536aea436b66ba not found: ID does not exist"
Sep 07 22:46:08.018540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.018510    2550 scope.go:117] "RemoveContainer" containerID="9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"
Sep 07 22:46:08.018746 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:46:08.018728    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5\": container with ID starting with 9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5 not found: ID does not exist" containerID="9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"
Sep 07 22:46:08.018799 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.018757    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5"} err="failed to get container status \"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5\": rpc error: code = NotFound desc = could not find container \"9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5\": container with ID starting with 9e4bcf23637422bdeb09ddfdc3f0b7f97577ed1ee90b9e39d0ee6e7888d094d5 not found: ID does not exist"
Sep 07 22:46:08.022150 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.022112    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:46:08.025422 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.025401    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e3b5bcb-openshift-default-d67c9c68-hlfzn"]
Sep 07 22:46:08.796940 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:08.796906    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" path="/var/lib/kubelet/pods/26394a83-25c2-4a07-8ac6-e80ce1b36946/volumes"
Sep 07 22:46:09.090191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:09.090086    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:46:09.090345 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:09.090209    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:46:09.102642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:09.102616    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:46:10.023963 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:10.023936    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:46:19.463012 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:19.462963    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:46:22.024310 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:22.024264    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:46:22.024712 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:22.024377    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:46:23.091546 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.091509    2550 generic.go:356] "Generic (PLEG): container finished" podID="e6c9955f-82ff-4290-a464-e5d605af995f" containerID="fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0" exitCode=0
Sep 07 22:46:23.092067 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.091584    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerDied","Data":"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"}
Sep 07 22:46:23.092067 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.091999    2550 scope.go:117] "RemoveContainer" containerID="fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"
Sep 07 22:46:23.093062 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.093036    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:23.093115 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.093099    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:23.093182 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:23.093147    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:46:24.105552 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:24.105514    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerStarted","Data":"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"}
Sep 07 22:46:24.633676 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:24.633643    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:24.633851 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:24.633801    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 07 22:46:24.633898 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:24.633857    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 07 22:46:25.634023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:25.633984    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 07 22:46:25.634532 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:25.634049    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 07 22:46:26.634081 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:26.634047    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 07 22:46:26.634492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:26.634103    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 07 22:46:27.633553 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:27.633513    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 07 22:46:27.633754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:27.633578    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 07 22:46:28.633453 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:28.633417    2550 patch_prober.go:28] interesting pod/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 07 22:46:28.633837 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:28.633474    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 07 22:46:29.637392 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:29.637363    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:29.637825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:29.637697    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:29.638421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:29.638402    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:31.336536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.336500    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"]
Sep 07 22:46:31.336941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.336927    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.336941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.336943    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.337018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.336956    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.337018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.336962    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.337082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.337024    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.337082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.337030    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="26394a83-25c2-4a07-8ac6-e80ce1b36946" containerName="istio-proxy"
Sep 07 22:46:31.340085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.340059    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:31.342648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.342578    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:46:31.342913 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.342651    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:46:31.346801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.346777    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"]
Sep 07 22:46:31.361637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.361612    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4\" (UID: \"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:31.462552 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.462525    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4\" (UID: \"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:31.463090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.463070    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4\" (UID: \"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:31.651927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.651846    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:31.920119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:31.920089    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"]
Sep 07 22:46:31.921656 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:46:31.921633    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fba1c5_3bb3_4b8d_92dd_a411a3cfdbba.slice/crio-8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9 WatchSource:0}: Error finding container 8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9: Status 404 returned error can't find the container with id 8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9
Sep 07 22:46:32.148471 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:32.148436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4" event={"ID":"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba","Type":"ContainerStarted","Data":"8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9"}
Sep 07 22:46:33.155078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:33.155039    2550 generic.go:356] "Generic (PLEG): container finished" podID="33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" containerID="2798831447a21461cab839d0ed06fa9ae9a6c24bed91bdd5aa3017c93d3facbb" exitCode=0
Sep 07 22:46:33.155515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:33.155122    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4" event={"ID":"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba","Type":"ContainerDied","Data":"2798831447a21461cab839d0ed06fa9ae9a6c24bed91bdd5aa3017c93d3facbb"}
Sep 07 22:46:34.293852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:34.293830    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:34.386338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:34.386309    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca\") pod \"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba\" (UID: \"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba\") "
Sep 07 22:46:34.386649 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:34.386626    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" (UID: "33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:34.487049 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:34.487026    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:35.167801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:35.167697    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4" event={"ID":"33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba","Type":"ContainerDied","Data":"8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9"}
Sep 07 22:46:35.167959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:35.167814    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dbd61dc185ce3676bca1bd7736dcebff38a0d5064b45777736433b307f31eb9"
Sep 07 22:46:35.167959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:35.167896    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"
Sep 07 22:46:37.915309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:37.915276    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.015561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.015528    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.096621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.096591    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"]
Sep 07 22:46:38.098169 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.098135    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.099078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.099056    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a602f13b-9lcm4"]
Sep 07 22:46:38.148976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.148950    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.221828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.221796    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:38.222027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.222000    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy" containerID="cri-o://7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860" gracePeriod=30
Sep 07 22:46:38.240844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.240803    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.652719 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.652637    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:38.796604 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.796575    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" path="/var/lib/kubelet/pods/33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba/volumes"
Sep 07 22:46:38.942536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:38.942502    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:39.040138 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:39.040097    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:40.872233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:40.872200    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:46:41.522486 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:41.522448    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:41.613795 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:41.613764    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:42.253698 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.253658    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:46:42.254272 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.254252    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" containerName="revoke-keys"
Sep 07 22:46:42.254331 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.254274    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" containerName="revoke-keys"
Sep 07 22:46:42.254371 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.254362    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fba1c5-3bb3-4b8d-92dd-a411a3cfdbba" containerName="revoke-keys"
Sep 07 22:46:42.258747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.258724    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.261318 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.261299    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7c3369\"/\"openshift-service-ca.crt\""
Sep 07 22:46:42.262462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.262431    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7c3369\"/\"autores-model-7c3369-kserve-self-signed-certs\""
Sep 07 22:46:42.262575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.262492    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7c3369\"/\"kube-root-ca.crt\""
Sep 07 22:46:42.262853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.262833    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7c3369\"/\"default-dockercfg-pbf84\""
Sep 07 22:46:42.266506 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.266113    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:46:42.350171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.350304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350176    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.350304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350278    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz79l\" (UniqueName: \"kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.350387 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350310    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.350387 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350332    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.350456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.350412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.451716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz79l\" (UniqueName: \"kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.451873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451726    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.451873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.451873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451805    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.451873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451845    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.452101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.451875    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.452184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.452112    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.452248 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.452200    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.452306 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.452251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.453891 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.453872    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.454332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.454312    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.459807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.459788    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz79l\" (UniqueName: \"kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l\") pod \"autores-model-7c3369-kserve-7c9b6b6447-fnw6z\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") " pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.564317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.564250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:42.569235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.569216    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:42.672350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.672299    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:42.771813 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.771777    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:42.843910 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:42.843885    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:46:42.845892 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:46:42.845867    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa96271e_73c9_4de5_b241_f5242eb4db80.slice/crio-0523aec5ba51c6dc09cb2afc18792e75d50ec773bd9872f296317456eb51ec9c WatchSource:0}: Error finding container 0523aec5ba51c6dc09cb2afc18792e75d50ec773bd9872f296317456eb51ec9c: Status 404 returned error can't find the container with id 0523aec5ba51c6dc09cb2afc18792e75d50ec773bd9872f296317456eb51ec9c
Sep 07 22:46:43.138570 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.138433    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:43.213943 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.213907    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" event={"ID":"fa96271e-73c9-4de5-b241-f5242eb4db80","Type":"ContainerStarted","Data":"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"}
Sep 07 22:46:43.213943 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.213949    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" event={"ID":"fa96271e-73c9-4de5-b241-f5242eb4db80","Type":"ContainerStarted","Data":"0523aec5ba51c6dc09cb2afc18792e75d50ec773bd9872f296317456eb51ec9c"}
Sep 07 22:46:43.232300 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.232254    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" podStartSLOduration=1.232239064 podStartE2EDuration="1.232239064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:43.230237404 +0000 UTC m=+883.072505669" watchObservedRunningTime="2026-09-07 22:46:43.232239064 +0000 UTC m=+883.074507331"
Sep 07 22:46:43.235441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.235417    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:43.315199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.315169    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:43.482526 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.482502    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:43.526470 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.526435    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:43.563436 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563410    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563466    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563508    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563529    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563564    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563586    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4pdw\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563610    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563642    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.563785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563707    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket\") pod \"e6c9955f-82ff-4290-a464-e5d605af995f\" (UID: \"e6c9955f-82ff-4290-a464-e5d605af995f\") "
Sep 07 22:46:43.564014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563820    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:43.564014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563876    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:43.564014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.563934    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:43.564227 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564059    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:43.564293 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564264    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.564293 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564286    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.564390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564301    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.564390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564315    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.564390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.564327    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:43.565864 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.565832    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:46:43.565970 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.565886    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw" (OuterVolumeSpecName: "kube-api-access-m4pdw") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "kube-api-access-m4pdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:43.566017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.565973    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:46:43.566017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.565991    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6c9955f-82ff-4290-a464-e5d605af995f" (UID: "e6c9955f-82ff-4290-a464-e5d605af995f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:43.665674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.665598    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c9955f-82ff-4290-a464-e5d605af995f-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.665674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.665625    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c9955f-82ff-4290-a464-e5d605af995f-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.665674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.665636    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pdw\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-kube-api-access-m4pdw\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.665674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.665645    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c9955f-82ff-4290-a464-e5d605af995f-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:43.665674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:43.665653    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c9955f-82ff-4290-a464-e5d605af995f-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:46:44.221537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.221496    2550 generic.go:356] "Generic (PLEG): container finished" podID="e6c9955f-82ff-4290-a464-e5d605af995f" containerID="7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860" exitCode=0
Sep 07 22:46:44.221722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.221554    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"
Sep 07 22:46:44.221722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.221584    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerDied","Data":"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"}
Sep 07 22:46:44.221722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.221633    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw" event={"ID":"e6c9955f-82ff-4290-a464-e5d605af995f","Type":"ContainerDied","Data":"d168404c8d422bb93d39f92fc70f7537ac005ed169fc89dcd80c4d8c8b9a937b"}
Sep 07 22:46:44.221722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.221657    2550 scope.go:117] "RemoveContainer" containerID="7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"
Sep 07 22:46:44.231804 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.231785    2550 scope.go:117] "RemoveContainer" containerID="fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"
Sep 07 22:46:44.239929 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.239910    2550 scope.go:117] "RemoveContainer" containerID="7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"
Sep 07 22:46:44.240175 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:46:44.240154    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860\": container with ID starting with 7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860 not found: ID does not exist" containerID="7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"
Sep 07 22:46:44.240250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.240182    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860"} err="failed to get container status \"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860\": rpc error: code = NotFound desc = could not find container \"7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860\": container with ID starting with 7b87b69b27a9c77e1e61486ee76df723fe363bd96028885d61912815371a7860 not found: ID does not exist"
Sep 07 22:46:44.240250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.240199    2550 scope.go:117] "RemoveContainer" containerID="fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"
Sep 07 22:46:44.240439 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:46:44.240421    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0\": container with ID starting with fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0 not found: ID does not exist" containerID="fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"
Sep 07 22:46:44.240480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.240444    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0"} err="failed to get container status \"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0\": rpc error: code = NotFound desc = could not find container \"fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0\": container with ID starting with fde30229ace0a1be982c2adc49716bdf95a6a0a2cfb3a004a4f8e5b683dd61a0 not found: ID does not exist"
Sep 07 22:46:44.243948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.243924    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:44.247251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.247229    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a602f13b-gw-openshift-default-55558b6696-bg5tw"]
Sep 07 22:46:44.316698 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.316673    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:44.802397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:44.802358    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" path="/var/lib/kubelet/pods/e6c9955f-82ff-4290-a464-e5d605af995f/volumes"
Sep 07 22:46:47.398812 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:47.398779    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:46:52.570263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:52.570209    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:52.570669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:52.570366    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:52.582961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:52.582933    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:53.288537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:53.288506    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:46:56.761896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:46:56.761861    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:47:07.159159 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159107    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"]
Sep 07 22:47:07.159533 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159493    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.159533 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159504    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.159533 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159517    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.159533 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159523    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.159662 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159597    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.159662 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.159606    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c9955f-82ff-4290-a464-e5d605af995f" containerName="istio-proxy"
Sep 07 22:47:07.163628 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.163611    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:07.167175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.167122    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:47:07.167306 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.167151    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:47:07.169591 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.169563    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"]
Sep 07 22:47:07.271741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.271713    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8\" (UID: \"f7c86657-adab-4760-b6fe-6c5ea52d670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:07.372511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.372476    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8\" (UID: \"f7c86657-adab-4760-b6fe-6c5ea52d670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:07.373042 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.373022    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8\" (UID: \"f7c86657-adab-4760-b6fe-6c5ea52d670d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:07.475088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.475059    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:07.744253 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:07.744229    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"]
Sep 07 22:47:08.367824 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:08.367787    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8" event={"ID":"f7c86657-adab-4760-b6fe-6c5ea52d670d","Type":"ContainerStarted","Data":"1d47e52220fb588ba71e93e0581f7ebd3448d222559269b7f67cf34b86f72b99"}
Sep 07 22:47:08.368229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:08.367854    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8" event={"ID":"f7c86657-adab-4760-b6fe-6c5ea52d670d","Type":"ContainerStarted","Data":"b887ef40a483f49707c7820383895c823545df590921b978460166083eae921c"}
Sep 07 22:47:08.382402 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:08.382357    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8" podStartSLOduration=1.38234217 podStartE2EDuration="1.38234217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:08.380950104 +0000 UTC m=+908.223218372" watchObservedRunningTime="2026-09-07 22:47:08.38234217 +0000 UTC m=+908.224610437"
Sep 07 22:47:09.374392 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:09.374354    2550 generic.go:356] "Generic (PLEG): container finished" podID="f7c86657-adab-4760-b6fe-6c5ea52d670d" containerID="1d47e52220fb588ba71e93e0581f7ebd3448d222559269b7f67cf34b86f72b99" exitCode=0
Sep 07 22:47:09.374392 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:09.374395    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8" event={"ID":"f7c86657-adab-4760-b6fe-6c5ea52d670d","Type":"ContainerDied","Data":"1d47e52220fb588ba71e93e0581f7ebd3448d222559269b7f67cf34b86f72b99"}
Sep 07 22:47:10.507389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:10.507359    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:10.602680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:10.602637    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca\") pod \"f7c86657-adab-4760-b6fe-6c5ea52d670d\" (UID: \"f7c86657-adab-4760-b6fe-6c5ea52d670d\") "
Sep 07 22:47:10.603009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:10.602983    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7c86657-adab-4760-b6fe-6c5ea52d670d" (UID: "f7c86657-adab-4760-b6fe-6c5ea52d670d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:10.703844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:10.703808    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c86657-adab-4760-b6fe-6c5ea52d670d-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:11.385649 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:11.385618    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"
Sep 07 22:47:11.385829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:11.385615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8" event={"ID":"f7c86657-adab-4760-b6fe-6c5ea52d670d","Type":"ContainerDied","Data":"b887ef40a483f49707c7820383895c823545df590921b978460166083eae921c"}
Sep 07 22:47:11.385829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:11.385766    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b887ef40a483f49707c7820383895c823545df590921b978460166083eae921c"
Sep 07 22:47:13.216344 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.216302    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:13.284075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.284038    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:13.382411 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.382370    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:13.401325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.401293    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"]
Sep 07 22:47:13.404094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.404068    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66774a4a-ft5x8"]
Sep 07 22:47:13.471141 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.471032    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:13.966772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:13.966733    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:47:14.471341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:14.471298    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:14.796647 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:14.796566    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c86657-adab-4760-b6fe-6c5ea52d670d" path="/var/lib/kubelet/pods/f7c86657-adab-4760-b6fe-6c5ea52d670d/volumes"
Sep 07 22:47:14.884923 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:14.884890    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:15.147246 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.147141    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:15.147633 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.147617    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c86657-adab-4760-b6fe-6c5ea52d670d" containerName="revoke-keys"
Sep 07 22:47:15.147685 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.147635    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c86657-adab-4760-b6fe-6c5ea52d670d" containerName="revoke-keys"
Sep 07 22:47:15.147721 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.147708    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c86657-adab-4760-b6fe-6c5ea52d670d" containerName="revoke-keys"
Sep 07 22:47:15.151080 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.151062    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.153640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.153616    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-01ae3268-openshift-default-dockercfg-ln8vq\""
Sep 07 22:47:15.160721 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.160699    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:15.167017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.166994    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:15.244545 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244511    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244555    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244583    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvrt\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244665    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244725    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.244776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244768    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.245015 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244838    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.245015 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.244865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346066    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346108    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346156    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346215    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346307    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346356    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346384    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346408    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvrt\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346595    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.346946 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.346923    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.347049 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.347020    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.347110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.347074    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.347422 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.347400    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.348979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.348945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.349397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.349378    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.353807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.353781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.353906 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.353889    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvrt\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt\") pod \"e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") " pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.374893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.374860    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:15.463960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.463917    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:15.531772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.531734    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:15.738846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.738813    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:15.740799 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:47:15.740760    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515bf750_8ec8_4021_a0f4_07794b0efd3f.slice/crio-0a18962ab9bb62ef9c65e8d55b18f5b22f211a8b105fc3b844ff28c985378861 WatchSource:0}: Error finding container 0a18962ab9bb62ef9c65e8d55b18f5b22f211a8b105fc3b844ff28c985378861: Status 404 returned error can't find the container with id 0a18962ab9bb62ef9c65e8d55b18f5b22f211a8b105fc3b844ff28c985378861
Sep 07 22:47:15.743063 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.743032    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:15.743149 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.743099    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:15.743192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:15.743145    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:16.417830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:16.417792    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerStarted","Data":"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"}
Sep 07 22:47:16.418001 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:16.417849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerStarted","Data":"0a18962ab9bb62ef9c65e8d55b18f5b22f211a8b105fc3b844ff28c985378861"}
Sep 07 22:47:16.440195 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:16.440115    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podStartSLOduration=1.440100989 podStartE2EDuration="1.440100989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:16.437917386 +0000 UTC m=+916.280185653" watchObservedRunningTime="2026-09-07 22:47:16.440100989 +0000 UTC m=+916.282369256"
Sep 07 22:47:16.464163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:16.464110    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:16.469229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:16.469204    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:17.423509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:17.423482    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:17.424554 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:17.424532    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:17.774336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:17.774301    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:17.872637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:17.872607    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:19.029513 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:19.029452    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:19.130667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:19.130627    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:19.343171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:19.343067    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:19.486044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:19.486014    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:19.584379 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:19.584345    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:25.446391 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:25.446355    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:26.013081 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:26.013025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:26.081151 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:26.081032    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:26.175356 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:26.175315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:26.229279 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:26.229246    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:26.538767 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:26.538740    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:31.524791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:31.524743    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:47:31.619967 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:31.619931    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:32.964104 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:32.964060    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:47:33.465295 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:33.465241    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:47:33.465504 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:33.465313    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:47:34.526766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.526727    2550 generic.go:356] "Generic (PLEG): container finished" podID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerID="b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096" exitCode=0
Sep 07 22:47:34.527217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.526797    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerDied","Data":"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"}
Sep 07 22:47:34.527267 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.527214    2550 scope.go:117] "RemoveContainer" containerID="b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"
Sep 07 22:47:34.528290 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.528263    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:34.528401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.528341    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:34.528401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:34.528389    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:47:35.533451 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:35.533409    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerStarted","Data":"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"}
Sep 07 22:47:36.464158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:36.464091    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:36.464574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:36.464542    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 07 22:47:36.464790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:36.464761    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 07 22:47:37.464941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:37.464454    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 07 22:47:37.464941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:37.464524    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 07 22:47:38.465594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:38.465092    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 07 22:47:38.465594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:38.465184    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 07 22:47:39.464791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:39.464742    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 07 22:47:39.465010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:39.464814    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 07 22:47:40.021599 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:40.021560    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:40.465326 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:40.465284    2550 patch_prober.go:28] interesting pod/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 07 22:47:40.465597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:40.465349    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 07 22:47:41.468689 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:41.468657    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:41.469160 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:41.468945    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:41.469594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:41.469578    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:42.573790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.573746    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"]
Sep 07 22:47:42.577426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.577408    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:42.580018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.579989    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:47:42.581195 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.581115    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:47:42.584366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.584339    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"]
Sep 07 22:47:42.611953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.611918    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-01ae3268-4wclv\" (UID: \"7784ecbc-c774-4453-8c3a-ba46068a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:42.713153 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.713089    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-01ae3268-4wclv\" (UID: \"7784ecbc-c774-4453-8c3a-ba46068a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:42.713857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.713831    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-01ae3268-4wclv\" (UID: \"7784ecbc-c774-4453-8c3a-ba46068a250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:42.890436 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:42.890353    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:43.164499 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:43.164472    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"]
Sep 07 22:47:43.166976 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:47:43.166947    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7784ecbc_c774_4453_8c3a_ba46068a250a.slice/crio-415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518 WatchSource:0}: Error finding container 415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518: Status 404 returned error can't find the container with id 415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518
Sep 07 22:47:43.601305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:43.601267    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv" event={"ID":"7784ecbc-c774-4453-8c3a-ba46068a250a","Type":"ContainerStarted","Data":"ae15a44e1dda1dfad9c8bbd34ac2b6254b9cd13d3b6be4014d365ee185a1c8bc"}
Sep 07 22:47:43.601688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:43.601325    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv" event={"ID":"7784ecbc-c774-4453-8c3a-ba46068a250a","Type":"ContainerStarted","Data":"415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518"}
Sep 07 22:47:43.616765 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:43.616709    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv" podStartSLOduration=1.6166918940000001 podStartE2EDuration="1.616691894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:43.614899717 +0000 UTC m=+943.457167988" watchObservedRunningTime="2026-09-07 22:47:43.616691894 +0000 UTC m=+943.458960160"
Sep 07 22:47:44.608841 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:44.608800    2550 generic.go:356] "Generic (PLEG): container finished" podID="7784ecbc-c774-4453-8c3a-ba46068a250a" containerID="ae15a44e1dda1dfad9c8bbd34ac2b6254b9cd13d3b6be4014d365ee185a1c8bc" exitCode=0
Sep 07 22:47:44.609232 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:44.608876    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv" event={"ID":"7784ecbc-c774-4453-8c3a-ba46068a250a","Type":"ContainerDied","Data":"ae15a44e1dda1dfad9c8bbd34ac2b6254b9cd13d3b6be4014d365ee185a1c8bc"}
Sep 07 22:47:45.752729 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:45.752704    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:45.842588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:45.842549    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca\") pod \"7784ecbc-c774-4453-8c3a-ba46068a250a\" (UID: \"7784ecbc-c774-4453-8c3a-ba46068a250a\") "
Sep 07 22:47:45.842942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:45.842921    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7784ecbc-c774-4453-8c3a-ba46068a250a" (UID: "7784ecbc-c774-4453-8c3a-ba46068a250a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:45.944199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:45.944168    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7784ecbc-c774-4453-8c3a-ba46068a250a-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.546785 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.546732    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" podUID="fa96271e-73c9-4de5-b241-f5242eb4db80" containerName="main" containerID="cri-o://188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1" gracePeriod=45
Sep 07 22:47:46.620783 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.620755    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"
Sep 07 22:47:46.620948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.620798    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv" event={"ID":"7784ecbc-c774-4453-8c3a-ba46068a250a","Type":"ContainerDied","Data":"415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518"}
Sep 07 22:47:46.620948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.620848    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415b7b2026497bd492b145a944bcdc5ad417b626f5c24eab0f08256a23028518"
Sep 07 22:47:46.813908 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.813882    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:47:46.853237 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853197    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz79l\" (UniqueName: \"kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853414 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853346    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853414 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853373    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853414 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853395    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853419    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853446    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs\") pod \"fa96271e-73c9-4de5-b241-f5242eb4db80\" (UID: \"fa96271e-73c9-4de5-b241-f5242eb4db80\") "
Sep 07 22:47:46.853688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853657    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:46.854059 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853888    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.854059 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853900    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:46.854059 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.853910    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home" (OuterVolumeSpecName: "home") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:46.855974 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.855943    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm" (OuterVolumeSpecName: "dshm") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:46.856093 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.855949    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:46.856093 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.856081    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l" (OuterVolumeSpecName: "kube-api-access-dz79l") pod "fa96271e-73c9-4de5-b241-f5242eb4db80" (UID: "fa96271e-73c9-4de5-b241-f5242eb4db80"). InnerVolumeSpecName "kube-api-access-dz79l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:46.954703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.954671    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.954703 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.954701    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.954898 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.954712    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa96271e-73c9-4de5-b241-f5242eb4db80-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.954898 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.954721    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa96271e-73c9-4de5-b241-f5242eb4db80-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:46.954898 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:46.954730    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz79l\" (UniqueName: \"kubernetes.io/projected/fa96271e-73c9-4de5-b241-f5242eb4db80-kube-api-access-dz79l\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:47.627431 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.627393    2550 generic.go:356] "Generic (PLEG): container finished" podID="fa96271e-73c9-4de5-b241-f5242eb4db80" containerID="188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1" exitCode=0
Sep 07 22:47:47.627624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.627444    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"
Sep 07 22:47:47.627624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.627476    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" event={"ID":"fa96271e-73c9-4de5-b241-f5242eb4db80","Type":"ContainerDied","Data":"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"}
Sep 07 22:47:47.627624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.627530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z" event={"ID":"fa96271e-73c9-4de5-b241-f5242eb4db80","Type":"ContainerDied","Data":"0523aec5ba51c6dc09cb2afc18792e75d50ec773bd9872f296317456eb51ec9c"}
Sep 07 22:47:47.627624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.627554    2550 scope.go:117] "RemoveContainer" containerID="188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"
Sep 07 22:47:47.637395 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.637376    2550 scope.go:117] "RemoveContainer" containerID="188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"
Sep 07 22:47:47.637685 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:47:47.637668    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1\": container with ID starting with 188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1 not found: ID does not exist" containerID="188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"
Sep 07 22:47:47.637733 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.637695    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1"} err="failed to get container status \"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1\": rpc error: code = NotFound desc = could not find container \"188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1\": container with ID starting with 188c8e0c0f8aca80d3fdea3f9f4000e687b62f6e928fc347f3a34dd55ddee2a1 not found: ID does not exist"
Sep 07 22:47:47.649185 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.649162    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:47:47.652697 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.652677    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7c3369/autores-model-7c3369-kserve-7c9b6b6447-fnw6z"]
Sep 07 22:47:47.770511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.770481    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:47:47.771009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.770993    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7784ecbc-c774-4453-8c3a-ba46068a250a" containerName="revoke-keys"
Sep 07 22:47:47.771075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.771012    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7784ecbc-c774-4453-8c3a-ba46068a250a" containerName="revoke-keys"
Sep 07 22:47:47.771075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.771023    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa96271e-73c9-4de5-b241-f5242eb4db80" containerName="main"
Sep 07 22:47:47.771075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.771029    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa96271e-73c9-4de5-b241-f5242eb4db80" containerName="main"
Sep 07 22:47:47.771228 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.771098    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7784ecbc-c774-4453-8c3a-ba46068a250a" containerName="revoke-keys"
Sep 07 22:47:47.771228 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.771108    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa96271e-73c9-4de5-b241-f5242eb4db80" containerName="main"
Sep 07 22:47:47.775531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.775512    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.778327 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.778302    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-08f59294-kserve-self-signed-certs\""
Sep 07 22:47:47.780925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.780905    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:47:47.862584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862557    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.862924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862589    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.862924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862610    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.862924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.862924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862739    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wgd\" (UniqueName: \"kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.863057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.862952    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.871038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.871014    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:47.964204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964164    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964268    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964300    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964395    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wgd\" (UniqueName: \"kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.964698 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.964579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.965418 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.965036    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.965926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.965833    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.968158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.968115    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.968367 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.968348    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:47.972552 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:47.972528    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wgd\" (UniqueName: \"kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd\") pod \"e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") " pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:48.085217 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.085172    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:48.355667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.355643    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:47:48.357879 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:47:48.357844    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc457fd6f_9638_42ba_92e1_9223eba1a980.slice/crio-c7d3cec61a20bca4a7129d56a8d0be470915282312f9a8748f903682e94ac64a WatchSource:0}: Error finding container c7d3cec61a20bca4a7129d56a8d0be470915282312f9a8748f903682e94ac64a: Status 404 returned error can't find the container with id c7d3cec61a20bca4a7129d56a8d0be470915282312f9a8748f903682e94ac64a
Sep 07 22:47:48.635725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.635631    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" event={"ID":"c457fd6f-9638-42ba-92e1-9223eba1a980","Type":"ContainerStarted","Data":"24c2366c0550a9930047b4cbb9a266910cbe1844a04735f932e2b06105fadb4c"}
Sep 07 22:47:48.635725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.635679    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" event={"ID":"c457fd6f-9638-42ba-92e1-9223eba1a980","Type":"ContainerStarted","Data":"c7d3cec61a20bca4a7129d56a8d0be470915282312f9a8748f903682e94ac64a"}
Sep 07 22:47:48.658181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.658105    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" podStartSLOduration=1.6580850900000002 podStartE2EDuration="1.65808509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:48.653489796 +0000 UTC m=+948.495758065" watchObservedRunningTime="2026-09-07 22:47:48.65808509 +0000 UTC m=+948.500353367"
Sep 07 22:47:48.796341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:48.796307    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa96271e-73c9-4de5-b241-f5242eb4db80" path="/var/lib/kubelet/pods/fa96271e-73c9-4de5-b241-f5242eb4db80/volumes"
Sep 07 22:47:49.419470 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.419429    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:49.519713 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.519655    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:49.599297 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.599263    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"]
Sep 07 22:47:49.601068 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.601040    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:49.601482 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.601462    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01ae3268-4wclv"]
Sep 07 22:47:49.654926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.654898    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:49.726894 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.726859    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:49.727371 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.727335    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy" containerID="cri-o://3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324" gracePeriod=30
Sep 07 22:47:49.749263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:49.749155    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:50.386191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:50.386120    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:50.486832 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:50.486798    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:50.584568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:50.584537    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:50.796852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:50.796817    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7784ecbc-c774-4453-8c3a-ba46068a250a" path="/var/lib/kubelet/pods/7784ecbc-c774-4453-8c3a-ba46068a250a/volumes"
Sep 07 22:47:51.879742 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:51.879698    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:54.370460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:54.370420    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:47:55.006076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.006019    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:55.123338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123307    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123383    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123401    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnvrt\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123420    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123457    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123559    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123617    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123658    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123709    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data\") pod \"515bf750-8ec8-4021-a0f4-07794b0efd3f\" (UID: \"515bf750-8ec8-4021-a0f4-07794b0efd3f\") "
Sep 07 22:47:55.123972 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123559    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:55.123972 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123746    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:55.123972 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123828    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:55.123972 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123866    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:55.124184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.123994    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:55.124184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.124136    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.124184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.124158    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.124184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.124171    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515bf750-8ec8-4021-a0f4-07794b0efd3f-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.124390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.124184    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.124390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.124204    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.125766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.125740    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:55.125766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.125752    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:55.126018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.125996    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt" (OuterVolumeSpecName: "kube-api-access-jnvrt") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "kube-api-access-jnvrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:55.126110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.126042    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "515bf750-8ec8-4021-a0f4-07794b0efd3f" (UID: "515bf750-8ec8-4021-a0f4-07794b0efd3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:47:55.224918 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.224891    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.224918 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.224916    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvrt\" (UniqueName: \"kubernetes.io/projected/515bf750-8ec8-4021-a0f4-07794b0efd3f-kube-api-access-jnvrt\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.225073 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.224926    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.225073 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.224934    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515bf750-8ec8-4021-a0f4-07794b0efd3f-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:47:55.676474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.676433    2550 generic.go:356] "Generic (PLEG): container finished" podID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerID="3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324" exitCode=0
Sep 07 22:47:55.676979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.676479    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"
Sep 07 22:47:55.676979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.676478    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerDied","Data":"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"}
Sep 07 22:47:55.676979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.676612    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68" event={"ID":"515bf750-8ec8-4021-a0f4-07794b0efd3f","Type":"ContainerDied","Data":"0a18962ab9bb62ef9c65e8d55b18f5b22f211a8b105fc3b844ff28c985378861"}
Sep 07 22:47:55.676979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.676635    2550 scope.go:117] "RemoveContainer" containerID="3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"
Sep 07 22:47:55.686457 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.686438    2550 scope.go:117] "RemoveContainer" containerID="b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"
Sep 07 22:47:55.701594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.701556    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:55.701829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.701805    2550 scope.go:117] "RemoveContainer" containerID="3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"
Sep 07 22:47:55.702409 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:47:55.702380    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324\": container with ID starting with 3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324 not found: ID does not exist" containerID="3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"
Sep 07 22:47:55.702487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.702422    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324"} err="failed to get container status \"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324\": rpc error: code = NotFound desc = could not find container \"3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324\": container with ID starting with 3f7dfdbfefb6e0b1455229b8ce5f273cfbd266be3322c4185969dd76da286324 not found: ID does not exist"
Sep 07 22:47:55.702487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.702447    2550 scope.go:117] "RemoveContainer" containerID="b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"
Sep 07 22:47:55.702742 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:47:55.702724    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096\": container with ID starting with b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096 not found: ID does not exist" containerID="b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"
Sep 07 22:47:55.702781 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.702748    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096"} err="failed to get container status \"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096\": rpc error: code = NotFound desc = could not find container \"b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096\": container with ID starting with b44acc606a283ea6650efac9e33b5775b7614b1f8eb1af4e4cacd136b867d096 not found: ID does not exist"
Sep 07 22:47:55.704039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:55.704025    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-01ae3268-openshift-default-55478f5cb-ssk68"]
Sep 07 22:47:56.796374 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:56.796336    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" path="/var/lib/kubelet/pods/515bf750-8ec8-4021-a0f4-07794b0efd3f/volumes"
Sep 07 22:47:57.531560 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:57.531521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:57.598236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:57.598198    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:57.678452 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:57.678413    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:57.779149 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:57.779088    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:57.894762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:57.894676    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:58.085590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:58.085551    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:58.085790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:58.085612    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:58.098149 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:58.098102    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:47:58.204420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:58.204388    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:47:58.706005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:47:58.705977    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:48:00.804161 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:00.804104    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:02.629639 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:02.629607    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:48:02.648396 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:02.648369    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:08.446205 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:08.446161    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:08.749750 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:08.749663    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:09.122142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.122049    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:09.373177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373068    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:48:09.373715 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373694    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:09.373790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373719    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:09.373790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373753    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:09.373790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373762    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:09.373896 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.373865    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:09.378454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.378426    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.381305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.381284    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-52b103-openshift-default-dockercfg-4xkdd\""
Sep 07 22:48:09.387061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.387035    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:48:09.396672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.396647    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:09.547346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547304    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fn6\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547352    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547371    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547490    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547547    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547569    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547595    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547679    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.547892 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.547700    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.648854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648777    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.648854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.648854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fn6\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648878    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648900    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648941    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.648997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649027    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649116 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649052    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649258    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649307    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649624 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649600    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649643    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.649694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.649680    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.651584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.651561    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.651668 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.651648    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.657843 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.657821    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.658102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.658083    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fn6\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6\") pod \"e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") " pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.677093 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.677068    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:09.691245 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.691223    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:09.969149 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.969111    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:48:09.971106 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:48:09.971078    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5ba38c_5f3d_46a1_9e58_75faa49fadc3.slice/crio-725682c4256b2b1e3cad584e5ca19a09cd2a6fce36cc4e2040321cd73c14eefb WatchSource:0}: Error finding container 725682c4256b2b1e3cad584e5ca19a09cd2a6fce36cc4e2040321cd73c14eefb: Status 404 returned error can't find the container with id 725682c4256b2b1e3cad584e5ca19a09cd2a6fce36cc4e2040321cd73c14eefb
Sep 07 22:48:09.973448 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.973416    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:09.973543 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.973483    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:09.973543 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:09.973511    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:10.763661 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:10.763618    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerStarted","Data":"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"}
Sep 07 22:48:10.764066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:10.763678    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerStarted","Data":"725682c4256b2b1e3cad584e5ca19a09cd2a6fce36cc4e2040321cd73c14eefb"}
Sep 07 22:48:10.781985 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:10.781921    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podStartSLOduration=1.7819050870000002 podStartE2EDuration="1.781905087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:48:10.780856735 +0000 UTC m=+970.623125028" watchObservedRunningTime="2026-09-07 22:48:10.781905087 +0000 UTC m=+970.624173353"
Sep 07 22:48:11.691753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:11.691710    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:11.696675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:11.696647    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:11.769425 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:11.769397    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:11.770421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:11.770399    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:12.651925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:12.651873    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:12.745612 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:12.745578    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:13.810606 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:13.810573    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:13.913746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:13.913711    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:14.107250 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:14.107169    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:14.264392 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:14.264350    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:14.364530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:14.364446    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:14.374844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:14.374808    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:48:17.650073 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:17.650029    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" podUID="c457fd6f-9638-42ba-92e1-9223eba1a980" containerName="main" containerID="cri-o://24c2366c0550a9930047b4cbb9a266910cbe1844a04735f932e2b06105fadb4c" gracePeriod=45
Sep 07 22:48:17.804887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:17.804855    2550 generic.go:356] "Generic (PLEG): container finished" podID="c457fd6f-9638-42ba-92e1-9223eba1a980" containerID="24c2366c0550a9930047b4cbb9a266910cbe1844a04735f932e2b06105fadb4c" exitCode=0
Sep 07 22:48:17.805040 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:17.804908    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" event={"ID":"c457fd6f-9638-42ba-92e1-9223eba1a980","Type":"ContainerDied","Data":"24c2366c0550a9930047b4cbb9a266910cbe1844a04735f932e2b06105fadb4c"}
Sep 07 22:48:17.908765 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:17.908710    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:48:18.023873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.023841    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.023884    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.023933    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.023957    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024021    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5wgd\" (UniqueName: \"kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024059    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir\") pod \"c457fd6f-9638-42ba-92e1-9223eba1a980\" (UID: \"c457fd6f-9638-42ba-92e1-9223eba1a980\") "
Sep 07 22:48:18.024278 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024234    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home" (OuterVolumeSpecName: "home") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:18.024393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024319    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache" (OuterVolumeSpecName: "model-cache") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:18.024393 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024335    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:18.024495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024469    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.024550 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024491    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.024550 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.024532    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.026216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.026190    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd" (OuterVolumeSpecName: "kube-api-access-k5wgd") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "kube-api-access-k5wgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:48:18.026363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.026340    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:48:18.026363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.026354    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm" (OuterVolumeSpecName: "dshm") pod "c457fd6f-9638-42ba-92e1-9223eba1a980" (UID: "c457fd6f-9638-42ba-92e1-9223eba1a980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:18.126004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.125969    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c457fd6f-9638-42ba-92e1-9223eba1a980-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.126004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.125999    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c457fd6f-9638-42ba-92e1-9223eba1a980-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.126004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.126011    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5wgd\" (UniqueName: \"kubernetes.io/projected/c457fd6f-9638-42ba-92e1-9223eba1a980-kube-api-access-k5wgd\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:18.810989 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.810960    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"
Sep 07 22:48:18.811417 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.810987    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn" event={"ID":"c457fd6f-9638-42ba-92e1-9223eba1a980","Type":"ContainerDied","Data":"c7d3cec61a20bca4a7129d56a8d0be470915282312f9a8748f903682e94ac64a"}
Sep 07 22:48:18.811417 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.811046    2550 scope.go:117] "RemoveContainer" containerID="24c2366c0550a9930047b4cbb9a266910cbe1844a04735f932e2b06105fadb4c"
Sep 07 22:48:18.826790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.826762    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:48:18.832102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:18.832075    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-08f59294-kserve-5879f959cd-b6sbn"]
Sep 07 22:48:20.796561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:20.796528    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c457fd6f-9638-42ba-92e1-9223eba1a980" path="/var/lib/kubelet/pods/c457fd6f-9638-42ba-92e1-9223eba1a980/volumes"
Sep 07 22:48:21.048354 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:21.048266    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:48:21.071468 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:21.071438    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:24.069597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:24.069493    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:29.669158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:29.669087    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:48:29.984401 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:29.984364    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:30.048331 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:30.048289    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:30.132269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:30.132231    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:30.231724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:30.231675    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:32.904057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.904018    2550 generic.go:356] "Generic (PLEG): container finished" podID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerID="318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe" exitCode=0
Sep 07 22:48:32.904531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.904090    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerDied","Data":"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"}
Sep 07 22:48:32.904531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.904491    2550 scope.go:117] "RemoveContainer" containerID="318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"
Sep 07 22:48:32.905538 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.905512    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:32.905628 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.905578    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:32.905628 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:32.905607    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:48:33.911686 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:33.911646    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerStarted","Data":"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"}
Sep 07 22:48:34.695770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:34.695710    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:34.696474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:34.696438    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 07 22:48:34.696596 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:34.696492    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 07 22:48:35.691815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:35.691778    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 07 22:48:35.692230 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:35.691835    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 07 22:48:36.071437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.071337    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" podUID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" containerName="main" containerID="cri-o://eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1" gracePeriod=45
Sep 07 22:48:36.375459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.375428    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:48:36.483758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483715    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.483956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483775    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.483956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483807    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8492\" (UniqueName: \"kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.483956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483827    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.483956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483866    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.483956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.483904    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs\") pod \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\" (UID: \"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab\") "
Sep 07 22:48:36.484275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.484073    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home" (OuterVolumeSpecName: "home") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:36.484275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.484080    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:36.484275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.484258    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:36.484437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.484283    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.484437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.484311    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.486001 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.485969    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm" (OuterVolumeSpecName: "dshm") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:36.486110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.486023    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:48:36.486110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.486067    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492" (OuterVolumeSpecName: "kube-api-access-b8492") pod "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" (UID: "1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab"). InnerVolumeSpecName "kube-api-access-b8492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:48:36.585020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.584930    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.585020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.584964    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8492\" (UniqueName: \"kubernetes.io/projected/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-kube-api-access-b8492\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.585020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.584978    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.585020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.584986    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:48:36.692096 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.692058    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 07 22:48:36.692512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.692118    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 07 22:48:36.930531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.930439    2550 generic.go:356] "Generic (PLEG): container finished" podID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" containerID="eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1" exitCode=0
Sep 07 22:48:36.930531 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.930484    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"
Sep 07 22:48:36.930732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.930531    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" event={"ID":"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab","Type":"ContainerDied","Data":"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"}
Sep 07 22:48:36.930732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.930573    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42" event={"ID":"1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab","Type":"ContainerDied","Data":"fe43c830a0c934156250e8ab1e7b82ca687634917f1237a6cedcae394d1dce0e"}
Sep 07 22:48:36.930732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.930594    2550 scope.go:117] "RemoveContainer" containerID="eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"
Sep 07 22:48:36.939728 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.939710    2550 scope.go:117] "RemoveContainer" containerID="eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"
Sep 07 22:48:36.939999 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:48:36.939981    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1\": container with ID starting with eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1 not found: ID does not exist" containerID="eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"
Sep 07 22:48:36.940045 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.940010    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1"} err="failed to get container status \"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1\": rpc error: code = NotFound desc = could not find container \"eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1\": container with ID starting with eaec328b212030118aaaaa6e0548bf4c63b3869b5af2c5234c32f464dc15b4b1 not found: ID does not exist"
Sep 07 22:48:36.948844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.948813    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:48:36.952494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:36.952458    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-08f59294-kserve-56c8ff986d-85p42"]
Sep 07 22:48:37.531047 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:37.531011    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:37.692292 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:37.692250    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 07 22:48:37.692768 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:37.692331    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 07 22:48:38.692177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:38.692111    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 07 22:48:38.692384 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:38.692194    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 07 22:48:38.797043 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:38.797004    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" path="/var/lib/kubelet/pods/1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab/volumes"
Sep 07 22:48:39.696369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:39.696339    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:39.696849 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:39.696724    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:39.697338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:39.697320    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:48:43.365155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:43.365105    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:48:45.978928 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:45.978887    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:51.713508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:51.713454    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:51.775729 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:51.775675    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:51.852143 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:51.852099    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:51.953145 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:51.953103    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:52.863959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.863922    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:48:52.864367 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864352    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c457fd6f-9638-42ba-92e1-9223eba1a980" containerName="main"
Sep 07 22:48:52.864412 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864370    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c457fd6f-9638-42ba-92e1-9223eba1a980" containerName="main"
Sep 07 22:48:52.864412 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864387    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" containerName="main"
Sep 07 22:48:52.864412 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864392    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" containerName="main"
Sep 07 22:48:52.864508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864471    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd7938f-d1ab-48d5-aa1a-bbb2f143e4ab" containerName="main"
Sep 07 22:48:52.864508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864480    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="515bf750-8ec8-4021-a0f4-07794b0efd3f" containerName="istio-proxy"
Sep 07 22:48:52.864508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.864488    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c457fd6f-9638-42ba-92e1-9223eba1a980" containerName="main"
Sep 07 22:48:52.867350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.867327    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.871229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.871200    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-79b9aa\"/\"test-model-79b9aa-kserve-self-signed-certs\""
Sep 07 22:48:52.871366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.871238    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-79b9aa\"/\"default-dockercfg-qftds\""
Sep 07 22:48:52.871366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.871250    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-79b9aa\"/\"openshift-service-ca.crt\""
Sep 07 22:48:52.871366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.871312    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-79b9aa\"/\"kube-root-ca.crt\""
Sep 07 22:48:52.879276 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.878122    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:48:52.891161 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.891111    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:48:52.935426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935381    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.935426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935425    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkc8r\" (UniqueName: \"kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.935669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935467    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.935669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935504    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.935669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935558    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:52.935669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:52.935626    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037100 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037065    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037184    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037501 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037338    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkc8r\" (UniqueName: \"kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037628 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037606    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037696 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037642    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.037934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.037909    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.040219 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.040164    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.040631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.040493    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.046084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.046059    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkc8r\" (UniqueName: \"kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r\") pod \"test-model-79b9aa-kserve-6c74c6cfc5-p4l4q\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") " pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.178609 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.178516    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:48:53.448675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:53.448643    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:48:53.450061 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:48:53.450041    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63b0868c_49b6_482f_8d5f_b663151e5282.slice/crio-182f4ea9d26169195a8f13e2fb13b9b856eca4226881ee1e56c295c47c42bea3 WatchSource:0}: Error finding container 182f4ea9d26169195a8f13e2fb13b9b856eca4226881ee1e56c295c47c42bea3: Status 404 returned error can't find the container with id 182f4ea9d26169195a8f13e2fb13b9b856eca4226881ee1e56c295c47c42bea3
Sep 07 22:48:54.027647 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:54.027607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" event={"ID":"63b0868c-49b6-482f-8d5f-b663151e5282","Type":"ContainerStarted","Data":"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"}
Sep 07 22:48:54.027647 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:54.027649    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" event={"ID":"63b0868c-49b6-482f-8d5f-b663151e5282","Type":"ContainerStarted","Data":"182f4ea9d26169195a8f13e2fb13b9b856eca4226881ee1e56c295c47c42bea3"}
Sep 07 22:48:54.043358 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:54.043307    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" podStartSLOduration=2.04329327 podStartE2EDuration="2.04329327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:48:54.042194443 +0000 UTC m=+1013.884462714" watchObservedRunningTime="2026-09-07 22:48:54.04329327 +0000 UTC m=+1013.885561536"
Sep 07 22:48:59.064024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:48:59.063989    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:49:02.366764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:02.366720    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:03.131562 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:03.131524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:03.179266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:03.179239    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:49:03.179266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:03.179277    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:49:03.192039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:03.192011    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:49:04.095649 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:04.095619    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:49:05.829701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:05.829666    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:05.896097 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:05.896066    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:05.943085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:05.943052    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:08.777805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:08.777774    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:08.883423 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:08.883393    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:08.978120 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:08.978082    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:09.078561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:09.078465    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:09.661450 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:09.661421    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:09.756721 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:09.756689    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:09.859106 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:09.859068    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:11.212493 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:11.212457    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:12.474986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:12.474948    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:49:16.902708 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:16.902662    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:16.962875 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:16.962848    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:17.566829 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:17.566791    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:19.068336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.068301    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:19.160522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.160488    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"]
Sep 07 22:49:19.165652 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.165626    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:19.168661 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.168639    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:49:19.169000 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.168647    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:49:19.169302 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.169019    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-52b103-mhc6f\" (UID: \"7f9ec795-a90c-4c61-a303-156ae6207150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:19.172813 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.172793    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"]
Sep 07 22:49:19.269905 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.269870    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-52b103-mhc6f\" (UID: \"7f9ec795-a90c-4c61-a303-156ae6207150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:19.270489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.270468    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-52b103-mhc6f\" (UID: \"7f9ec795-a90c-4c61-a303-156ae6207150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:19.478498 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.478461    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:19.748087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:19.748057    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"]
Sep 07 22:49:19.749908 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:49:19.749877    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9ec795_a90c_4c61_a303_156ae6207150.slice/crio-6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b WatchSource:0}: Error finding container 6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b: Status 404 returned error can't find the container with id 6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b
Sep 07 22:49:20.173467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:20.173422    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f" event={"ID":"7f9ec795-a90c-4c61-a303-156ae6207150","Type":"ContainerStarted","Data":"fe14f6f6f0a4b7fc3ef79618884d08511c35bbd8c52bdded0fbf5d9062c891f2"}
Sep 07 22:49:20.173835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:20.173479    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f" event={"ID":"7f9ec795-a90c-4c61-a303-156ae6207150","Type":"ContainerStarted","Data":"6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b"}
Sep 07 22:49:20.188543 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:20.188499    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f" podStartSLOduration=1.188483854 podStartE2EDuration="1.188483854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:20.186569285 +0000 UTC m=+1040.028837555" watchObservedRunningTime="2026-09-07 22:49:20.188483854 +0000 UTC m=+1040.030752120"
Sep 07 22:49:21.180440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:21.180405    2550 generic.go:356] "Generic (PLEG): container finished" podID="7f9ec795-a90c-4c61-a303-156ae6207150" containerID="fe14f6f6f0a4b7fc3ef79618884d08511c35bbd8c52bdded0fbf5d9062c891f2" exitCode=0
Sep 07 22:49:21.180845 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:21.180470    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f" event={"ID":"7f9ec795-a90c-4c61-a303-156ae6207150","Type":"ContainerDied","Data":"fe14f6f6f0a4b7fc3ef79618884d08511c35bbd8c52bdded0fbf5d9062c891f2"}
Sep 07 22:49:22.324428 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:22.324399    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:22.391989 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:22.391955    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca\") pod \"7f9ec795-a90c-4c61-a303-156ae6207150\" (UID: \"7f9ec795-a90c-4c61-a303-156ae6207150\") "
Sep 07 22:49:22.392341 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:22.392319    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f9ec795-a90c-4c61-a303-156ae6207150" (UID: "7f9ec795-a90c-4c61-a303-156ae6207150"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:49:22.493074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:22.493044    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9ec795-a90c-4c61-a303-156ae6207150-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:23.191284 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.191256    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"
Sep 07 22:49:23.191443 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.191283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f" event={"ID":"7f9ec795-a90c-4c61-a303-156ae6207150","Type":"ContainerDied","Data":"6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b"}
Sep 07 22:49:23.191443 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.191325    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b8fc15e60779f4f5c807abd43c35f1e2b7ab21456c7868e925032db514a386b"
Sep 07 22:49:23.283083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.283051    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:23.383628 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.383594    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:23.560866 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.560780    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:23.794236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.794199    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:49:23.801803 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.795354    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9ec795-a90c-4c61-a303-156ae6207150" containerName="revoke-keys"
Sep 07 22:49:23.801803 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.795385    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9ec795-a90c-4c61-a303-156ae6207150" containerName="revoke-keys"
Sep 07 22:49:23.801803 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.795633    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9ec795-a90c-4c61-a303-156ae6207150" containerName="revoke-keys"
Sep 07 22:49:23.804494 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.804464    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.807004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.806984    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2872fd39-openshift-default-dockercfg-zpxpn\""
Sep 07 22:49:23.811088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.811028    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:49:23.814239 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.814206    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:23.903606 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903565    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903621    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htdf\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903687    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903791 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903751    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903802    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903829    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.903925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:23.904074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:23.903928    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004644 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004608    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004644 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004651    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004681    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004717    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004734    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.004876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.004764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005053    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005078    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005153    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005162    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htdf\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005331    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005382    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.005496 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.005444    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.007177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.007157    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.007599 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.007580    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.015529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.015505    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.015790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.015769    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htdf\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf\") pod \"e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") " pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.074686 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.074595    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:24.117746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.117714    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:24.394700 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.394668    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:49:24.396138 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:49:24.396098    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cd4744_1beb_40d2_8f41_25ee884c28be.slice/crio-f252ce10032ed9130590cf724c1210640db98a298dfaae4e2f7062a3a8cce157 WatchSource:0}: Error finding container f252ce10032ed9130590cf724c1210640db98a298dfaae4e2f7062a3a8cce157: Status 404 returned error can't find the container with id f252ce10032ed9130590cf724c1210640db98a298dfaae4e2f7062a3a8cce157
Sep 07 22:49:24.398379 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.398344    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:24.398485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.398412    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:24.398485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:24.398440    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:25.205839 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:25.205799    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerStarted","Data":"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"}
Sep 07 22:49:25.206041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:25.205849    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerStarted","Data":"f252ce10032ed9130590cf724c1210640db98a298dfaae4e2f7062a3a8cce157"}
Sep 07 22:49:25.226757 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:25.226708    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podStartSLOduration=2.226695096 podStartE2EDuration="2.226695096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:25.224435271 +0000 UTC m=+1045.066703538" watchObservedRunningTime="2026-09-07 22:49:25.226695096 +0000 UTC m=+1045.068963469"
Sep 07 22:49:26.118141 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:26.118106    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:26.123451 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:26.123424    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:26.211473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:26.211450    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:26.212314 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:26.212293    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:27.093743 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:27.093677    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:27.197053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:27.197023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:28.233107 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:28.233073    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:28.343248 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:28.343214    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:28.444971 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:28.444932    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:28.808701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:28.808644    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:28.908429 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:28.908385    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:29.689308 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.689271    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:29.758251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.758213    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:29.858322 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.858285    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:29.875284 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.875257    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"]
Sep 07 22:49:29.879303 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.879274    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-52b103-mhc6f"]
Sep 07 22:49:29.958053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:29.957957    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:30.225301 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:30.225207    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:30.796807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:30.796769    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f9ec795-a90c-4c61-a303-156ae6207150" path="/var/lib/kubelet/pods/7f9ec795-a90c-4c61-a303-156ae6207150/volumes"
Sep 07 22:49:31.263945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:31.263908    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:49:35.266965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:35.266932    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:36.247027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.246984    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:36.309254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.309217    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:36.463711 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.463680    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:49:36.463919 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.463894    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" containerID="cri-o://75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49" gracePeriod=30
Sep 07 22:49:36.478778 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.478747    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:36.921264 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:36.921228    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:37.213105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:37.213073    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:37.690620 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:37.690588    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:38.562675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:38.562640    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:38.656437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:38.656407    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:38.756335 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:38.756301    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:39.699363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:39.699329    2550 patch_prober.go:28] interesting pod/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:49:39.699523 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:39.699396    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:49:41.738640 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.738579    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:49:41.874115 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874070    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874319 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874175    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874319 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874281    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874324    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874350    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874393    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data" (OuterVolumeSpecName: "istio-data") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:41.874460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874416    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874453    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9fn6\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874492    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874533    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:41.874741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874548    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket\") pod \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\" (UID: \"be5ba38c-5f3d-46a1-9e58-75faa49fadc3\") "
Sep 07 22:49:41.874741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874592    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:41.874741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874726    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:41.875014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874869    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:49:41.875014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874913    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.875014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874930    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.875014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874943    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.875014 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.874958    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.876520 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.876494    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:41.876660 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.876635    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:49:41.876948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.876930    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6" (OuterVolumeSpecName: "kube-api-access-k9fn6") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "kube-api-access-k9fn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:41.877007 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.876948    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token" (OuterVolumeSpecName: "istio-token") pod "be5ba38c-5f3d-46a1-9e58-75faa49fadc3" (UID: "be5ba38c-5f3d-46a1-9e58-75faa49fadc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:41.976087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.976051    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.976087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.976080    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.976087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.976089    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.976333 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.976100    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fn6\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-kube-api-access-k9fn6\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:41.976333 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:41.976108    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5ba38c-5f3d-46a1-9e58-75faa49fadc3-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:49:42.305904 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.305862    2550 generic.go:356] "Generic (PLEG): container finished" podID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerID="75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49" exitCode=0
Sep 07 22:49:42.306076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.305908    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"
Sep 07 22:49:42.306076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.305949    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerDied","Data":"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"}
Sep 07 22:49:42.306076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.306008    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj" event={"ID":"be5ba38c-5f3d-46a1-9e58-75faa49fadc3","Type":"ContainerDied","Data":"725682c4256b2b1e3cad584e5ca19a09cd2a6fce36cc4e2040321cd73c14eefb"}
Sep 07 22:49:42.306076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.306043    2550 scope.go:117] "RemoveContainer" containerID="75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"
Sep 07 22:49:42.316229 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.316207    2550 scope.go:117] "RemoveContainer" containerID="318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"
Sep 07 22:49:42.324731 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.324708    2550 scope.go:117] "RemoveContainer" containerID="75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"
Sep 07 22:49:42.325003 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:49:42.324986    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49\": container with ID starting with 75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49 not found: ID does not exist" containerID="75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"
Sep 07 22:49:42.325045 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.325011    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49"} err="failed to get container status \"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49\": rpc error: code = NotFound desc = could not find container \"75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49\": container with ID starting with 75a093bc07d422179719a69d024a8976957f34b2d565c1e0dd9c008e5f969e49 not found: ID does not exist"
Sep 07 22:49:42.325045 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.325026    2550 scope.go:117] "RemoveContainer" containerID="318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"
Sep 07 22:49:42.325277 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:49:42.325261    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe\": container with ID starting with 318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe not found: ID does not exist" containerID="318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"
Sep 07 22:49:42.325325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.325284    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe"} err="failed to get container status \"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe\": rpc error: code = NotFound desc = could not find container \"318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe\": container with ID starting with 318c2160c5279209dcfa77a2dfdb693f2317b41d440bc4c38c20a40dc9093dbe not found: ID does not exist"
Sep 07 22:49:42.329275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.329250    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:49:42.331337 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.331317    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-52b103-openshift-default-b59cfdd84-hmprj"]
Sep 07 22:49:42.796698 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:42.796665    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" path="/var/lib/kubelet/pods/be5ba38c-5f3d-46a1-9e58-75faa49fadc3/volumes"
Sep 07 22:49:47.467501 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:47.467460    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:49:49.347236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.347206    2550 generic.go:356] "Generic (PLEG): container finished" podID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerID="eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3" exitCode=0
Sep 07 22:49:49.347662 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.347281    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerDied","Data":"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"}
Sep 07 22:49:49.347662 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.347636    2550 scope.go:117] "RemoveContainer" containerID="eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"
Sep 07 22:49:49.348713 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.348685    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:49.348766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.348753    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:49.348805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:49.348794    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:49:50.358522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:50.358479    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerStarted","Data":"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"}
Sep 07 22:49:51.117931 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:51.117901    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:51.118105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:51.118078    2550 patch_prober.go:28] interesting pod/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:49:51.118171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:51.118152    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 07 22:49:52.006094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:52.006063    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:52.119165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:52.119106    2550 patch_prober.go:28] interesting pod/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:49:52.119332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:52.119201    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 07 22:49:52.642066 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:52.642036    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:52.742389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:52.742353    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:53.029315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:53.029283    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:53.118719 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:53.118680    2550 patch_prober.go:28] interesting pod/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:49:53.118886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:53.118737    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 07 22:49:53.484755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:53.484723    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:54.118826 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:54.118788    2550 patch_prober.go:28] interesting pod/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:49:54.119244 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:54.118862    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 07 22:49:55.118911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:55.118877    2550 patch_prober.go:28] interesting pod/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:49:55.119334 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:55.118940    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 07 22:49:55.989063 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:55.989034    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:56.102693 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.102663    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:56.122205 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.122183    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:56.122558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.122445    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:56.123165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.123149    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:49:56.316961 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.316879    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:56.410119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:56.410091    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:58.881092 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:58.881044    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:58.981266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:58.981236    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:49:59.209170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209121    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"]
Sep 07 22:49:59.209567 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209551    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:49:59.209620 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209569    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:49:59.209620 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209577    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:49:59.209620 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209583    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:49:59.209716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.209664    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:49:59.212823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.212806    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:49:59.215541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.215522    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:49:59.215641 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.215554    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:49:59.222674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.222652    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"]
Sep 07 22:49:59.341723 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.341688    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4\" (UID: \"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:49:59.443076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.443047    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4\" (UID: \"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:49:59.443631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.443613    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4\" (UID: \"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:49:59.525281 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.525206    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:49:59.798502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:49:59.798469    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"]
Sep 07 22:49:59.800014 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:49:59.799989    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa41bc24_5716_4ca7_8cb5_3aa707aeb6e4.slice/crio-b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05 WatchSource:0}: Error finding container b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05: Status 404 returned error can't find the container with id b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05
Sep 07 22:50:00.427431 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:00.427388    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4" event={"ID":"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4","Type":"ContainerStarted","Data":"9725cc0ec045bad44ae32312e6071d8b8b300d041a31efe35dfb2aaade7046fe"}
Sep 07 22:50:00.427809 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:00.427452    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4" event={"ID":"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4","Type":"ContainerStarted","Data":"b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05"}
Sep 07 22:50:00.441320 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:00.441273    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4" podStartSLOduration=1.4412564190000001 podStartE2EDuration="1.441256419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:00.440492116 +0000 UTC m=+1080.282760384" watchObservedRunningTime="2026-09-07 22:50:00.441256419 +0000 UTC m=+1080.283524685"
Sep 07 22:50:01.434270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:01.434224    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:01.435273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:01.435251    2550 generic.go:356] "Generic (PLEG): container finished" podID="aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" containerID="9725cc0ec045bad44ae32312e6071d8b8b300d041a31efe35dfb2aaade7046fe" exitCode=0
Sep 07 22:50:01.435363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:01.435338    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4" event={"ID":"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4","Type":"ContainerDied","Data":"9725cc0ec045bad44ae32312e6071d8b8b300d041a31efe35dfb2aaade7046fe"}
Sep 07 22:50:02.578976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:02.578948    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:50:02.772866 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:02.772780    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca\") pod \"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4\" (UID: \"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4\") "
Sep 07 22:50:02.773142 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:02.773107    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" (UID: "aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:02.874176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:02.874145    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:03.447660 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:03.447631    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"
Sep 07 22:50:03.447874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:03.447673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4" event={"ID":"aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4","Type":"ContainerDied","Data":"b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05"}
Sep 07 22:50:03.447874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:03.447722    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6507a81335733941d7a6505ccd778119306146a5fb3122523979222fa310b05"
Sep 07 22:50:09.677368 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.677326    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:09.777074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.777036    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:09.873563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.873524    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"]
Sep 07 22:50:09.876708 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.876679    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:09.877192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.877174    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2872fd39-cpsj4"]
Sep 07 22:50:09.930214 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:09.930154    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:10.213510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:10.213476    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:10.796519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:10.796487    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" path="/var/lib/kubelet/pods/aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4/volumes"
Sep 07 22:50:11.395573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:11.395531    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:12.490100 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:12.490066    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:50:12.510881 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:12.510839    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:12.801555 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:12.801483    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:14.268397 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:14.268358    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:50:15.879173 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:15.879136    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:15.978301 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:15.978248    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:16.096937 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.096902    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:50:16.097215 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.097183    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy" containerID="cri-o://c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e" gracePeriod=30
Sep 07 22:50:16.111101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.111078    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:16.432846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.432811    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:16.535350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.535318    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:16.650815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:16.650775    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:21.351758 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.351733    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:50:21.421764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421696    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.421900 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421763    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.421900 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421790    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.421900 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421831    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.421900 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421868    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.421900 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421892    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.422170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421941    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.422170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.421990    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:21.422170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422004    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2htdf\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.422170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422031    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token\") pod \"b4cd4744-1beb-40d2-8f41-25ee884c28be\" (UID: \"b4cd4744-1beb-40d2-8f41-25ee884c28be\") "
Sep 07 22:50:21.422170 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422106    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:21.422455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422327    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:21.422455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422113    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:21.422455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422397    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:21.422455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422449    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.422616 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422467    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.422616 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422478    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.422616 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.422488    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.424760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.424736    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:21.424876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.424754    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:21.424876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.424743    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:50:21.424876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.424780    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf" (OuterVolumeSpecName: "kube-api-access-2htdf") pod "b4cd4744-1beb-40d2-8f41-25ee884c28be" (UID: "b4cd4744-1beb-40d2-8f41-25ee884c28be"). InnerVolumeSpecName "kube-api-access-2htdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:21.523305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.523272    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cd4744-1beb-40d2-8f41-25ee884c28be-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.523305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.523303    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.523305 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.523312    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.523480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.523320    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htdf\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-kube-api-access-2htdf\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.523480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.523331    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cd4744-1beb-40d2-8f41-25ee884c28be-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:21.556725 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.556700    2550 generic.go:356] "Generic (PLEG): container finished" podID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerID="c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e" exitCode=0
Sep 07 22:50:21.556814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.556748    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"
Sep 07 22:50:21.556814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.556784    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerDied","Data":"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"}
Sep 07 22:50:21.556886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.556837    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b" event={"ID":"b4cd4744-1beb-40d2-8f41-25ee884c28be","Type":"ContainerDied","Data":"f252ce10032ed9130590cf724c1210640db98a298dfaae4e2f7062a3a8cce157"}
Sep 07 22:50:21.556886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.556862    2550 scope.go:117] "RemoveContainer" containerID="c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"
Sep 07 22:50:21.566090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.566072    2550 scope.go:117] "RemoveContainer" containerID="eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"
Sep 07 22:50:21.577835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.577819    2550 scope.go:117] "RemoveContainer" containerID="c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"
Sep 07 22:50:21.578794 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:50:21.578769    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e\": container with ID starting with c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e not found: ID does not exist" containerID="c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"
Sep 07 22:50:21.578956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.578924    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e"} err="failed to get container status \"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e\": rpc error: code = NotFound desc = could not find container \"c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e\": container with ID starting with c3eda8df50b543f07a3c49eed351d0d8319ca2944335c50a96c2afa0549e3c0e not found: ID does not exist"
Sep 07 22:50:21.579076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.578958    2550 scope.go:117] "RemoveContainer" containerID="eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"
Sep 07 22:50:21.579772 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:50:21.579745    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3\": container with ID starting with eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3 not found: ID does not exist" containerID="eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"
Sep 07 22:50:21.579864 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.579781    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3"} err="failed to get container status \"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3\": rpc error: code = NotFound desc = could not find container \"eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3\": container with ID starting with eb6455087360acafed121d443be76b4bdc1a7124f29571daec911ff342faffb3 not found: ID does not exist"
Sep 07 22:50:21.580238 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.580220    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:50:21.582307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:21.582287    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2872fd39-openshift-default-7bb4dfd856-bb75b"]
Sep 07 22:50:22.398867 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:22.398837    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:22.500065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:22.500025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:22.599187 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:22.599160    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:22.717163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:22.717123    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:22.795819 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:22.795791    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" path="/var/lib/kubelet/pods/b4cd4744-1beb-40d2-8f41-25ee884c28be/volumes"
Sep 07 22:50:23.129766 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.129691    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:23.199865 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.199837    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:23.262073 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.262042    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:23.343273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.343240    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:23.406938 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.406862    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:23.673780 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:23.673681    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:25.270726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:25.270691    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:50:27.512927 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.512878    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" podUID="63b0868c-49b6-482f-8d5f-b663151e5282" containerName="main" containerID="cri-o://ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2" gracePeriod=45
Sep 07 22:50:27.768694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.768637    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:50:27.875903 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.875875    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.875911    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.875985    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876017    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkc8r\" (UniqueName: \"kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876062    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876093    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir\") pod \"63b0868c-49b6-482f-8d5f-b663151e5282\" (UID: \"63b0868c-49b6-482f-8d5f-b663151e5282\") "
Sep 07 22:50:27.876790 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876757    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home" (OuterVolumeSpecName: "home") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:27.876899 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876772    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache" (OuterVolumeSpecName: "model-cache") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:27.876899 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.876834    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:27.878442 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.878419    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:27.878542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.878525    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm" (OuterVolumeSpecName: "dshm") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:27.878596 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.878528    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r" (OuterVolumeSpecName: "kube-api-access-jkc8r") pod "63b0868c-49b6-482f-8d5f-b663151e5282" (UID: "63b0868c-49b6-482f-8d5f-b663151e5282"). InnerVolumeSpecName "kube-api-access-jkc8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:27.978013 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.977991    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkc8r\" (UniqueName: \"kubernetes.io/projected/63b0868c-49b6-482f-8d5f-b663151e5282-kube-api-access-jkc8r\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.978013 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.978013    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.978180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.978023    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.978180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.978031    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b0868c-49b6-482f-8d5f-b663151e5282-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.978180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.978038    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:27.978180 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:27.978046    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b0868c-49b6-482f-8d5f-b663151e5282-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:28.441175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.441120    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:28.543358 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.543325    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:28.599389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.599356    2550 generic.go:356] "Generic (PLEG): container finished" podID="63b0868c-49b6-482f-8d5f-b663151e5282" containerID="ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2" exitCode=0
Sep 07 22:50:28.599510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.599406    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"
Sep 07 22:50:28.599510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.599444    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" event={"ID":"63b0868c-49b6-482f-8d5f-b663151e5282","Type":"ContainerDied","Data":"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"}
Sep 07 22:50:28.599510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.599487    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q" event={"ID":"63b0868c-49b6-482f-8d5f-b663151e5282","Type":"ContainerDied","Data":"182f4ea9d26169195a8f13e2fb13b9b856eca4226881ee1e56c295c47c42bea3"}
Sep 07 22:50:28.599510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.599505    2550 scope.go:117] "RemoveContainer" containerID="ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"
Sep 07 22:50:28.608933 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.608916    2550 scope.go:117] "RemoveContainer" containerID="ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"
Sep 07 22:50:28.609206 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:50:28.609183    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2\": container with ID starting with ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2 not found: ID does not exist" containerID="ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"
Sep 07 22:50:28.609286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.609215    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2"} err="failed to get container status \"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2\": rpc error: code = NotFound desc = could not find container \"ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2\": container with ID starting with ad7b577da4038a312c827a1e8d22862fd64690f1e468a2fe61f4351021260ca2 not found: ID does not exist"
Sep 07 22:50:28.621018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.620997    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:50:28.627203 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.627183    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-79b9aa/test-model-79b9aa-kserve-6c74c6cfc5-p4l4q"]
Sep 07 22:50:28.796239 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:28.796164    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b0868c-49b6-482f-8d5f-b663151e5282" path="/var/lib/kubelet/pods/63b0868c-49b6-482f-8d5f-b663151e5282/volumes"
Sep 07 22:50:29.324316 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:29.324276    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:29.623364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:29.623288    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:29.782292 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:29.782251    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:29.842152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:29.842109    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:29.910592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:29.910510    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:30.023558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.023527    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:30.336145 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336095    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"]
Sep 07 22:50:30.336592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336572    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:30.336592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336594    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:30.336746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336634    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b0868c-49b6-482f-8d5f-b663151e5282" containerName="main"
Sep 07 22:50:30.336746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336643    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b0868c-49b6-482f-8d5f-b663151e5282" containerName="main"
Sep 07 22:50:30.336746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336662    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" containerName="revoke-keys"
Sep 07 22:50:30.336746 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336671    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" containerName="revoke-keys"
Sep 07 22:50:30.336955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336759    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5ba38c-5f3d-46a1-9e58-75faa49fadc3" containerName="istio-proxy"
Sep 07 22:50:30.336955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336778    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:30.336955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336787    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:30.336955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336799    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa41bc24-5716-4ca7-8cb5-3aa707aeb6e4" containerName="revoke-keys"
Sep 07 22:50:30.336955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.336809    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b0868c-49b6-482f-8d5f-b663151e5282" containerName="main"
Sep 07 22:50:30.341470 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.341448    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:30.346308 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.346283    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:50:30.346308 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.346301    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:50:30.348216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.348190    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"]
Sep 07 22:50:30.402589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.402499    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6\" (UID: \"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:30.432735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.432706    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:30.503433 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.503396    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6\" (UID: \"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:30.503984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.503963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6\" (UID: \"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:30.654204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.654103    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:30.923512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:30.923485    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"]
Sep 07 22:50:30.925194 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:50:30.925162    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9281dba_5bc7_4430_8ffa_4c199cbcb0b2.slice/crio-8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd WatchSource:0}: Error finding container 8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd: Status 404 returned error can't find the container with id 8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd
Sep 07 22:50:31.621567 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:31.621538    2550 generic.go:356] "Generic (PLEG): container finished" podID="a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" containerID="b8c8126a9fafec4a101168f1c49c491e4b36a21385b7e99013535f6f97cb66ca" exitCode=0
Sep 07 22:50:31.621710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:31.621632    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6" event={"ID":"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2","Type":"ContainerDied","Data":"b8c8126a9fafec4a101168f1c49c491e4b36a21385b7e99013535f6f97cb66ca"}
Sep 07 22:50:31.621710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:31.621691    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6" event={"ID":"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2","Type":"ContainerStarted","Data":"8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd"}
Sep 07 22:50:32.458027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.457992    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:32.550851 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.550823    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:32.767644 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.767612    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:32.823226 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.823194    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca\") pod \"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2\" (UID: \"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2\") "
Sep 07 22:50:32.823518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.823497    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" (UID: "a9281dba-5bc7-4430-8ffa-4c199cbcb0b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:32.924184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:32.924119    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:33.622833 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.622790    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:33.635329 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.635295    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6" event={"ID":"a9281dba-5bc7-4430-8ffa-4c199cbcb0b2","Type":"ContainerDied","Data":"8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd"}
Sep 07 22:50:33.635477 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.635361    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8213e46c1c5cff57b1911073c491e9402bd6d1299e6d9ccb001b575c44ba8dcd"
Sep 07 22:50:33.635477 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.635308    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"
Sep 07 22:50:33.722807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.722772    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:33.991176 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:33.991140    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:34.187233 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:34.187202    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:34.289091 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:34.289009    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:37.872912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:37.872877    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:50:38.208174 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:38.208147    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:40.451375 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:40.451334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:41.103706 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.103672    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:41.172741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.172708    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:41.228691 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.228656    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:41.286472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.286436    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"]
Sep 07 22:50:41.288519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.288499    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79b9aa-ztdz6"]
Sep 07 22:50:41.292492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.292464    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:41.630090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:41.630053    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:42.796011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:42.795979    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" path="/var/lib/kubelet/pods/a9281dba-5bc7-4430-8ffa-4c199cbcb0b2/volumes"
Sep 07 22:50:47.402592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:47.402551    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:47.461064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:47.461017    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:47.635517 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:47.635483    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:50:47.635772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:47.635746    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" containerID="cri-o://1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07" gracePeriod=30
Sep 07 22:50:47.659158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:47.659080    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:48.205801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.205765    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:48.423463 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.423408    2550 patch_prober.go:28] interesting pod/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:50:48.423905 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.423501    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:50:48.470996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.470912    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 22:50:48.471801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.471771    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:48.471801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.471799    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cd4744-1beb-40d2-8f41-25ee884c28be" containerName="istio-proxy"
Sep 07 22:50:48.471955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.471811    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" containerName="revoke-keys"
Sep 07 22:50:48.471955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.471820    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" containerName="revoke-keys"
Sep 07 22:50:48.471955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.471918    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9281dba-5bc7-4430-8ffa-4c199cbcb0b2" containerName="revoke-keys"
Sep 07 22:50:48.475263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.475238    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.478224 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.478202    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1d0395-openshift-default-dockercfg-dcw77\""
Sep 07 22:50:48.491035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.491011    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 22:50:48.502736 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.502707    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:48.553371 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553333    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjjq\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553410    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553437    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553463    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553669    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553750    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.553850 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.553810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654443    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654485    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654513    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654547    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654575    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjjq\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654605    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654626    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.654753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654652    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.655112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654930    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.655112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.654949    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.655112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.655013    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.655416 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.655393    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.655756 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.655733    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.657030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.657005    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.657420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.657403    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.662220 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.662184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.662348 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.662322    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjjq\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq\") pod \"e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") " pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.787370 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.787289    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:48.969805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:48.969767    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:49.066088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.066063    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 22:50:49.067320 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:50:49.067294    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9649c969_f473_4263_8fb1_acbcd941fe07.slice/crio-0fdefcae3b1f6acd25dc1afc523422d3a2ea4cc2d548099a7a5fa97fda6a7816 WatchSource:0}: Error finding container 0fdefcae3b1f6acd25dc1afc523422d3a2ea4cc2d548099a7a5fa97fda6a7816: Status 404 returned error can't find the container with id 0fdefcae3b1f6acd25dc1afc523422d3a2ea4cc2d548099a7a5fa97fda6a7816
Sep 07 22:50:49.069935 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.069900    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:49.070036 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.069980    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:49.070036 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.070023    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:49.727443 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.727408    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerStarted","Data":"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"}
Sep 07 22:50:49.727875 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.727455    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerStarted","Data":"0fdefcae3b1f6acd25dc1afc523422d3a2ea4cc2d548099a7a5fa97fda6a7816"}
Sep 07 22:50:49.746885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.746840    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podStartSLOduration=1.74682848 podStartE2EDuration="1.74682848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:49.745537511 +0000 UTC m=+1129.587805790" watchObservedRunningTime="2026-09-07 22:50:49.74682848 +0000 UTC m=+1129.589096747"
Sep 07 22:50:49.787396 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:49.787369    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:50.801747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:50.801717    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:51.624256 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:51.624215    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:51.728911 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:51.728851    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:51.739104 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:51.739078    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:51.740091 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:51.740072    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:50:52.674086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.674050    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:50:52.789456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.789418    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:52.888920 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.888888    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:52.895669 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.895646    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:50:52.990086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990060    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990106    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990156    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990186    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctd2j\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990221    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990289 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990247    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990298    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990330    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990362    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy\") pod \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\" (UID: \"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8\") "
Sep 07 22:50:52.990577 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990517    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:52.990802 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990751    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:52.990862 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990827    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:52.990955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990931    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:52.991017 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.990984    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:52.991078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.991028    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:52.992456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.992415    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j" (OuterVolumeSpecName: "kube-api-access-ctd2j") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "kube-api-access-ctd2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:52.992456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.992435    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:52.992747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.992725    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:52.992810 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:52.992728    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" (UID: "7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:50:53.091849 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091826    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091854    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091865    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091873    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091881    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091888    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctd2j\" (UniqueName: \"kubernetes.io/projected/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-kube-api-access-ctd2j\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091898    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.091959 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.091906    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:50:53.124413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.124381    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:53.263557 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.263468    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:53.357882 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.357848    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:53.751415 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.751379    2550 generic.go:356] "Generic (PLEG): container finished" podID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerID="1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07" exitCode=0
Sep 07 22:50:53.751840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.751427    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"
Sep 07 22:50:53.751840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.751467    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerDied","Data":"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"}
Sep 07 22:50:53.751840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.751524    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz" event={"ID":"7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8","Type":"ContainerDied","Data":"c1210b9a95e0a312a80ef9f72a6115e2f74fe8cd0e8b38157ccbb350b8e13327"}
Sep 07 22:50:53.751840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.751551    2550 scope.go:117] "RemoveContainer" containerID="1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"
Sep 07 22:50:53.761613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.761594    2550 scope.go:117] "RemoveContainer" containerID="65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.771721    2550 scope.go:117] "RemoveContainer" containerID="1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:50:53.772093    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07\": container with ID starting with 1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07 not found: ID does not exist" containerID="1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.772141    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07"} err="failed to get container status \"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07\": rpc error: code = NotFound desc = could not find container \"1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07\": container with ID starting with 1dcf16d3c08cbf5ec4a9c16ec64b1182194d5245f130c211b3c6fd5ff1601b07 not found: ID does not exist"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.772162    2550 scope.go:117] "RemoveContainer" containerID="65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:50:53.772445    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4\": container with ID starting with 65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4 not found: ID does not exist" containerID="65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"
Sep 07 22:50:53.773255 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.772468    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4"} err="failed to get container status \"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4\": rpc error: code = NotFound desc = could not find container \"65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4\": container with ID starting with 65f32316c129ccc6abea7b087d99edad962f0362783884ad3367b4809d574ec4 not found: ID does not exist"
Sep 07 22:50:53.774663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.774643    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:50:53.776762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:53.776743    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-79b9aa-openshift-default-7dbc758ffd-h62mz"]
Sep 07 22:50:54.600974 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:54.600930    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:54.796519 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:54.796482    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" path="/var/lib/kubelet/pods/7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8/volumes"
Sep 07 22:50:55.381945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:55.381914    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:56.111845 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:56.111810    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:57.351854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.351817    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:57.661581 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.661489    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:57.998474 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998441    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:50:57.998894 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998876    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:57.998894 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998892    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:57.999061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998902    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:57.999061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998908    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:57.999061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998983    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:57.999061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:57.998995    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e11d1a1-c014-45c1-8e8a-cac7ff8b2ee8" containerName="istio-proxy"
Sep 07 22:50:58.003769 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.003749    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.006701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.006678    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5483b1-openshift-default-dockercfg-tq7l2\""
Sep 07 22:50:58.015603 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.015580    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:50:58.043350 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.043327    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:58.135313 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135280    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135338    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135389    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135421    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135534    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5d9q\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.135625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.135573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236857    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5d9q\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236930    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236959    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.236999 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.236977    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237306 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237315    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237403 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237340    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237534 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237510    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237604 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237554    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.237684 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.237666    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.239158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.239117    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.239650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.239631    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:50:58.239786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.239766    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.244403 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.244378    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5d9q\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.244495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.244422    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token\") pod \"e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") " pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.315354 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.315276    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:58.591158 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.591113    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:50:58.592877 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:50:58.592849    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf55f1c_1011_4e7f_bded_6050fe0ffc9b.slice/crio-4bd0f12c482fe2f8db9344aedaea01cb19c792a8fcf3ddeabc6dd7c4b648edac WatchSource:0}: Error finding container 4bd0f12c482fe2f8db9344aedaea01cb19c792a8fcf3ddeabc6dd7c4b648edac: Status 404 returned error can't find the container with id 4bd0f12c482fe2f8db9344aedaea01cb19c792a8fcf3ddeabc6dd7c4b648edac
Sep 07 22:50:58.595094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.595061    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:58.595218 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.595166    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:58.595218 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.595212    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:50:58.782304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.782265    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerStarted","Data":"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"}
Sep 07 22:50:58.782486 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.782333    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerStarted","Data":"4bd0f12c482fe2f8db9344aedaea01cb19c792a8fcf3ddeabc6dd7c4b648edac"}
Sep 07 22:50:58.800089 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:58.799804    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podStartSLOduration=1.799787598 podStartE2EDuration="1.799787598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:58.799119977 +0000 UTC m=+1138.641388258" watchObservedRunningTime="2026-09-07 22:50:58.799787598 +0000 UTC m=+1138.642055865"
Sep 07 22:50:59.315879 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:59.315847    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:59.320554 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:59.320529    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:59.788179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:59.788154    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:50:59.789028 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:50:59.789007    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:51:00.653152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:00.651461    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:00.743750 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:00.743712    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:01.909201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:01.909168    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:02.009513 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:02.009479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:02.109686 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:02.109649    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:02.380367 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:02.380334    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:51:02.384325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:02.384294    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:02.458151 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:02.458005    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:07.780381 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:07.780337    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:51:07.780796 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:07.780427    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:51:09.859027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.857290    2550 generic.go:356] "Generic (PLEG): container finished" podID="9649c969-f473-4263-8fb1-acbcd941fe07" containerID="509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d" exitCode=0
Sep 07 22:51:09.859027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.857450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerDied","Data":"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"}
Sep 07 22:51:09.859027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.857892    2550 scope.go:117] "RemoveContainer" containerID="509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"
Sep 07 22:51:09.861029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.860692    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:09.861029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.860788    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:09.861029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:09.860835    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:10.869206 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:10.869163    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerStarted","Data":"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"}
Sep 07 22:51:11.092690 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:11.092649    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:51:11.787681 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:11.787634    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:51:11.787681 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:11.787661    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 07 22:51:11.787969 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:11.787722    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 07 22:51:12.788185 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:12.788102    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 07 22:51:12.788683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:12.788207    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 07 22:51:13.788332 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:13.788293    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 07 22:51:13.788810 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:13.788353    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 07 22:51:14.712699 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:14.712664    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:14.787883 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:14.787844    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 07 22:51:14.788068 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:14.787906    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 07 22:51:15.790738 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:15.790692    2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 07 22:51:15.791190 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:15.790772    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 07 22:51:15.831251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:15.831222    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:51:15.831390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:15.831267    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:51:16.564224 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:16.564178    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:16.865045 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:16.865008    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:51:16.866514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:16.865674    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:51:16.867296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:16.867269    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 22:51:16.889102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:16.889079    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:17.159802 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.159706    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 22:51:17.163057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.163032    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.165508 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.165482    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7823cc-openshift-default-dockercfg-sbh77\""
Sep 07 22:51:17.174883 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.174797    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 22:51:17.183331 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.183306    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:17.219113 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219086    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219121    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219215    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219251    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219315    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219449 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219401    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219579 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219444    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfjnh\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.219579 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.219520    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.320782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320745    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.320782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320789    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320846    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320870    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320895    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320928    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320955    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.320987    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjnh\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.321293    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321421 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.321371    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.321454    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321629 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.321606    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.321762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.321741    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.323273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.323255    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.323571 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.323551    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.328704 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.328682    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.328951 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.328931    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjnh\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh\") pod \"e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") " pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.474947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.474914    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:17.749342 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.749254    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:17.755192 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.755165    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 22:51:17.755968 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:17.755944    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385b048f_c008_42a9_811f_b9de2c5d8735.slice/crio-9cc9d657bea515cfb5db30ebfc4d04e4128ffd135cfc59cb89e158ae28c43b07 WatchSource:0}: Error finding container 9cc9d657bea515cfb5db30ebfc4d04e4128ffd135cfc59cb89e158ae28c43b07: Status 404 returned error can't find the container with id 9cc9d657bea515cfb5db30ebfc4d04e4128ffd135cfc59cb89e158ae28c43b07
Sep 07 22:51:17.758184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.758153    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.758293 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.758227    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.758293 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.758279    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.923817 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.923775    2550 generic.go:356] "Generic (PLEG): container finished" podID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerID="00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16" exitCode=0
Sep 07 22:51:17.924249 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.923823    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerDied","Data":"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"}
Sep 07 22:51:17.924394 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.924290    2550 scope.go:117] "RemoveContainer" containerID="00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"
Sep 07 22:51:17.925504 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.925475    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.925621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.925563    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.925621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.925610    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:17.925994 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.925972    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerStarted","Data":"19f1aead48959b59e6580364e18d5fbdcc23e591f5f77bd67faae04956c67862"}
Sep 07 22:51:17.926083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.926014    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerStarted","Data":"9cc9d657bea515cfb5db30ebfc4d04e4128ffd135cfc59cb89e158ae28c43b07"}
Sep 07 22:51:17.965764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:17.965706    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podStartSLOduration=0.96568546 podStartE2EDuration="965.68546ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:17.963694162 +0000 UTC m=+1157.805962432" watchObservedRunningTime="2026-09-07 22:51:17.96568546 +0000 UTC m=+1157.807953725"
Sep 07 22:51:18.476188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:18.476110    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:18.481815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:18.481786    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:18.933267 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:18.933177    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerStarted","Data":"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"}
Sep 07 22:51:18.933670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:18.933555    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:18.934538 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:18.934513    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:19.315476 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:19.315438    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:51:19.315683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:19.315661    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 22:51:19.315735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:19.315710    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 22:51:19.832223 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:19.832184    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:19.932658 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:19.932621    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:20.316471 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:20.316429    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 22:51:20.316869 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:20.316498    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 22:51:21.014050 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.014018    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:21.117486 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.117450    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:21.214646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.214613    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:21.316514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.316420    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 22:51:21.316514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.316483    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 22:51:21.483578 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.483537    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:21.576748 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:21.576668    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:22.316443 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:22.316405    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 22:51:22.316627 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:22.316463    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 22:51:22.836530 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:22.836497    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:23.316261 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:23.316220    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 22:51:23.316451 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:23.316295    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 22:51:23.443814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:23.443786    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:24.320227 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:24.320197    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:51:24.320579 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:24.320558    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:51:24.321100 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:24.321082    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:51:25.593573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:25.593536    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:26.205814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.205775    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:26.210726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.210704    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:26.213378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.213356    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b689c\""
Sep 07 22:51:26.215046 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.215024    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:26.309736 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.309704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9bq\" (UniqueName: \"kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq\") pod \"test-curl-7578-140678379788016\" (UID: \"c4fb0779-ee5b-4078-b7b9-747e460587f8\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:26.410943 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.410904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9bq\" (UniqueName: \"kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq\") pod \"test-curl-7578-140678379788016\" (UID: \"c4fb0779-ee5b-4078-b7b9-747e460587f8\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:26.420369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.420342    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9bq\" (UniqueName: \"kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq\") pod \"test-curl-7578-140678379788016\" (UID: \"c4fb0779-ee5b-4078-b7b9-747e460587f8\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:26.521309 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.521234    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:26.794022 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:26.793976    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4fb0779_ee5b_4078_b7b9_747e460587f8.slice/crio-f81523c432a9577a52799c3a9a77c085121d0b9d6119c4de40bda504e16e1f3c WatchSource:0}: Error finding container f81523c432a9577a52799c3a9a77c085121d0b9d6119c4de40bda504e16e1f3c: Status 404 returned error can't find the container with id f81523c432a9577a52799c3a9a77c085121d0b9d6119c4de40bda504e16e1f3c
Sep 07 22:51:26.795385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.795365    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:51:26.797168 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.797115    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:26.989189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:26.989157    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788016" event={"ID":"c4fb0779-ee5b-4078-b7b9-747e460587f8","Type":"ContainerStarted","Data":"f81523c432a9577a52799c3a9a77c085121d0b9d6119c4de40bda504e16e1f3c"}
Sep 07 22:51:27.080251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:27.080179    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:51:29.004948 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:29.004906    2550 generic.go:356] "Generic (PLEG): container finished" podID="c4fb0779-ee5b-4078-b7b9-747e460587f8" containerID="d7ff17ff2badf3ef456192292ed88f164650696c48ff932c3f3919ea5d41d86a" exitCode=0
Sep 07 22:51:29.005351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:29.004996    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788016" event={"ID":"c4fb0779-ee5b-4078-b7b9-747e460587f8","Type":"ContainerDied","Data":"d7ff17ff2badf3ef456192292ed88f164650696c48ff932c3f3919ea5d41d86a"}
Sep 07 22:51:30.179942 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.179915    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.202708 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.202680    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7578-140678379788016_c4fb0779-ee5b-4078-b7b9-747e460587f8/test-curl-7578-140678379788016/0.log"
Sep 07 22:51:30.223320 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.223291    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:30.226454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.226426    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:30.266646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.266549    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz9bq\" (UniqueName: \"kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq\") pod \"c4fb0779-ee5b-4078-b7b9-747e460587f8\" (UID: \"c4fb0779-ee5b-4078-b7b9-747e460587f8\") "
Sep 07 22:51:30.272088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.272052    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq" (OuterVolumeSpecName: "kube-api-access-wz9bq") pod "c4fb0779-ee5b-4078-b7b9-747e460587f8" (UID: "c4fb0779-ee5b-4078-b7b9-747e460587f8"). InnerVolumeSpecName "kube-api-access-wz9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:51:30.345660 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.345628    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:30.346348 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.346321    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fb0779-ee5b-4078-b7b9-747e460587f8" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:30.346348 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.346349    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fb0779-ee5b-4078-b7b9-747e460587f8" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:30.346518 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.346473    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fb0779-ee5b-4078-b7b9-747e460587f8" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:30.349772 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.349750    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.353511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.353477    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:30.368702 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.368673    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9bq\" (UniqueName: \"kubernetes.io/projected/c4fb0779-ee5b-4078-b7b9-747e460587f8-kube-api-access-wz9bq\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:51:30.470382 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.470338    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrrc\" (UniqueName: \"kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc\") pod \"test-curl-7578-140678379788016\" (UID: \"58929992-d06f-4e89-af30-b04bd7fd6307\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.571147 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.571033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrrc\" (UniqueName: \"kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc\") pod \"test-curl-7578-140678379788016\" (UID: \"58929992-d06f-4e89-af30-b04bd7fd6307\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.578883 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.578856    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrrc\" (UniqueName: \"kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc\") pod \"test-curl-7578-140678379788016\" (UID: \"58929992-d06f-4e89-af30-b04bd7fd6307\") " pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.669715 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.669675    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:30.798181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.798146    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fb0779-ee5b-4078-b7b9-747e460587f8" path="/var/lib/kubelet/pods/c4fb0779-ee5b-4078-b7b9-747e460587f8/volumes"
Sep 07 22:51:30.939996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:30.939965    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:30.941433 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:30.941408    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58929992_d06f_4e89_af30_b04bd7fd6307.slice/crio-4ed3272bbffba3726fe8d29515995ba442422376a09a1d71399807b6fd49123e WatchSource:0}: Error finding container 4ed3272bbffba3726fe8d29515995ba442422376a09a1d71399807b6fd49123e: Status 404 returned error can't find the container with id 4ed3272bbffba3726fe8d29515995ba442422376a09a1d71399807b6fd49123e
Sep 07 22:51:31.016478 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:31.016441    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788016" event={"ID":"58929992-d06f-4e89-af30-b04bd7fd6307","Type":"ContainerStarted","Data":"4ed3272bbffba3726fe8d29515995ba442422376a09a1d71399807b6fd49123e"}
Sep 07 22:51:31.017775 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:31.017753    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:31.017893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:31.017798    2550 scope.go:117] "RemoveContainer" containerID="d7ff17ff2badf3ef456192292ed88f164650696c48ff932c3f3919ea5d41d86a"
Sep 07 22:51:32.027672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:32.027581    2550 generic.go:356] "Generic (PLEG): container finished" podID="58929992-d06f-4e89-af30-b04bd7fd6307" containerID="ab36c9e22f8e3abe6d49996dd96dcaa749c15d9495a837c804f94958ea1185ce" exitCode=0
Sep 07 22:51:32.027672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:32.027635    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788016" event={"ID":"58929992-d06f-4e89-af30-b04bd7fd6307","Type":"ContainerDied","Data":"ab36c9e22f8e3abe6d49996dd96dcaa749c15d9495a837c804f94958ea1185ce"}
Sep 07 22:51:33.226755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.226711    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:33.244188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.244163    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7578-140678379788016_58929992-d06f-4e89-af30-b04bd7fd6307/test-curl-7578-140678379788016/0.log"
Sep 07 22:51:33.276548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.276516    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:33.280010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.279985    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-140678379788016"]
Sep 07 22:51:33.390090 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.390002    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:33.390712 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.390690    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58929992-d06f-4e89-af30-b04bd7fd6307" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:33.390771 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.390716    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="58929992-d06f-4e89-af30-b04bd7fd6307" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:33.390821 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.390809    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="58929992-d06f-4e89-af30-b04bd7fd6307" containerName="test-curl-7578-140678379788016"
Sep 07 22:51:33.394390 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.394369    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:33.397103 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.397074    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktrrc\" (UniqueName: \"kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc\") pod \"58929992-d06f-4e89-af30-b04bd7fd6307\" (UID: \"58929992-d06f-4e89-af30-b04bd7fd6307\") "
Sep 07 22:51:33.398912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.398885    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:33.403338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.403271    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc" (OuterVolumeSpecName: "kube-api-access-ktrrc") pod "58929992-d06f-4e89-af30-b04bd7fd6307" (UID: "58929992-d06f-4e89-af30-b04bd7fd6307"). InnerVolumeSpecName "kube-api-access-ktrrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:51:33.498174 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.498111    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrb4\" (UniqueName: \"kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4\") pod \"test-curl-7578-140678379788592\" (UID: \"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:33.498357 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.498283    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrrc\" (UniqueName: \"kubernetes.io/projected/58929992-d06f-4e89-af30-b04bd7fd6307-kube-api-access-ktrrc\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:51:33.599603 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.599564    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrb4\" (UniqueName: \"kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4\") pod \"test-curl-7578-140678379788592\" (UID: \"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:33.607094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.607073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrb4\" (UniqueName: \"kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4\") pod \"test-curl-7578-140678379788592\" (UID: \"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:33.715968 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:33.715930    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:34.004827 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:34.004796    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:34.006142 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:34.006103    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2a41c2_5103_43cd_8ef4_3183f8fc5e48.slice/crio-4a0766ac3b050d5d45c32cb8d7c2d01e2139810703b56ef921ed6f60de41bcf3 WatchSource:0}: Error finding container 4a0766ac3b050d5d45c32cb8d7c2d01e2139810703b56ef921ed6f60de41bcf3: Status 404 returned error can't find the container with id 4a0766ac3b050d5d45c32cb8d7c2d01e2139810703b56ef921ed6f60de41bcf3
Sep 07 22:51:34.041528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:34.041505    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:34.041625 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:34.041540    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed3272bbffba3726fe8d29515995ba442422376a09a1d71399807b6fd49123e"
Sep 07 22:51:34.042776 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:34.042753    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788592" event={"ID":"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48","Type":"ContainerStarted","Data":"4a0766ac3b050d5d45c32cb8d7c2d01e2139810703b56ef921ed6f60de41bcf3"}
Sep 07 22:51:34.066879 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:51:34.066842    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7578-140678379788016\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="58929992-d06f-4e89-af30-b04bd7fd6307" pod="opendatahub/test-curl-7578-140678379788016"
Sep 07 22:51:34.797561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:34.797523    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58929992-d06f-4e89-af30-b04bd7fd6307" path="/var/lib/kubelet/pods/58929992-d06f-4e89-af30-b04bd7fd6307/volumes"
Sep 07 22:51:35.050885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:35.050793    2550 generic.go:356] "Generic (PLEG): container finished" podID="fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" containerID="769f9d6f0950d3c3e936cfc1eaa8697c69dd61b16c1daf45151c25df4e799472" exitCode=0
Sep 07 22:51:35.050885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:35.050850    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788592" event={"ID":"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48","Type":"ContainerDied","Data":"769f9d6f0950d3c3e936cfc1eaa8697c69dd61b16c1daf45151c25df4e799472"}
Sep 07 22:51:36.191897 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.191866    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.209819 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.209797    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7578-140678379788592_fd2a41c2-5103-43cd-8ef4-3183f8fc5e48/test-curl-7578-140678379788592/0.log"
Sep 07 22:51:36.234570 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.234536    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:36.236549 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.236527    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:36.326537 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.326454    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nrb4\" (UniqueName: \"kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4\") pod \"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48\" (UID: \"fd2a41c2-5103-43cd-8ef4-3183f8fc5e48\") "
Sep 07 22:51:36.328491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.328465    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4" (OuterVolumeSpecName: "kube-api-access-9nrb4") pod "fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" (UID: "fd2a41c2-5103-43cd-8ef4-3183f8fc5e48"). InnerVolumeSpecName "kube-api-access-9nrb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:51:36.382555 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.382519    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:36.382945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.382932    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:36.383011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.382947    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:36.383011 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.383008    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:36.385988 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.385972    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.391657 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.391623    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:36.427573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.427533    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nrb4\" (UniqueName: \"kubernetes.io/projected/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48-kube-api-access-9nrb4\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:51:36.528618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.528577    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpd4\" (UniqueName: \"kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4\") pod \"test-curl-7578-140678379788592\" (UID: \"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.629491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.629405    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpd4\" (UniqueName: \"kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4\") pod \"test-curl-7578-140678379788592\" (UID: \"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.637141 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.637100    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpd4\" (UniqueName: \"kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4\") pod \"test-curl-7578-140678379788592\" (UID: \"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57\") " pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.696388 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.696356    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:36.799079 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.799045    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2a41c2-5103-43cd-8ef4-3183f8fc5e48" path="/var/lib/kubelet/pods/fd2a41c2-5103-43cd-8ef4-3183f8fc5e48/volumes"
Sep 07 22:51:36.966515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:36.966482    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:36.968599 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:36.968569    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4fc06e2_8440_4a10_8a1c_4d31e06e1e57.slice/crio-229c4db72a95fadf21b895cf38c76e7bfe9fe8a08f52167a5062c70c36235513 WatchSource:0}: Error finding container 229c4db72a95fadf21b895cf38c76e7bfe9fe8a08f52167a5062c70c36235513: Status 404 returned error can't find the container with id 229c4db72a95fadf21b895cf38c76e7bfe9fe8a08f52167a5062c70c36235513
Sep 07 22:51:37.063441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:37.063414    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:37.063609 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:37.063459    2550 scope.go:117] "RemoveContainer" containerID="769f9d6f0950d3c3e936cfc1eaa8697c69dd61b16c1daf45151c25df4e799472"
Sep 07 22:51:37.064922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:37.064898    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788592" event={"ID":"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57","Type":"ContainerStarted","Data":"229c4db72a95fadf21b895cf38c76e7bfe9fe8a08f52167a5062c70c36235513"}
Sep 07 22:51:38.072578 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.072543    2550 generic.go:356] "Generic (PLEG): container finished" podID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" containerID="e461b6cbe94b87bb8a4687f6621fdc851773121d73a3d4cc2f1bcd822458bd55" exitCode=0
Sep 07 22:51:38.073060 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.072620    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-140678379788592" event={"ID":"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57","Type":"ContainerDied","Data":"e461b6cbe94b87bb8a4687f6621fdc851773121d73a3d4cc2f1bcd822458bd55"}
Sep 07 22:51:38.074163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.074116    2550 generic.go:356] "Generic (PLEG): container finished" podID="385b048f-c008-42a9-811f-b9de2c5d8735" containerID="19f1aead48959b59e6580364e18d5fbdcc23e591f5f77bd67faae04956c67862" exitCode=0
Sep 07 22:51:38.074163 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.074150    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerDied","Data":"19f1aead48959b59e6580364e18d5fbdcc23e591f5f77bd67faae04956c67862"}
Sep 07 22:51:38.074514 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.074500    2550 scope.go:117] "RemoveContainer" containerID="19f1aead48959b59e6580364e18d5fbdcc23e591f5f77bd67faae04956c67862"
Sep 07 22:51:38.075528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.075504    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:38.075612 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.075570    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:38.075612 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:38.075601    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:39.081916 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.081878    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerStarted","Data":"30482e9ffedd3bc8708d36411254dd77ca4fc1341715a7057c2663e8da3fc881"}
Sep 07 22:51:39.224140 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.224107    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:39.243197 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.243165    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7578-140678379788592_b4fc06e2-8440-4a10-8a1c-4d31e06e1e57/test-curl-7578-140678379788592/0.log"
Sep 07 22:51:39.267953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.267269    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbpd4\" (UniqueName: \"kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4\") pod \"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57\" (UID: \"b4fc06e2-8440-4a10-8a1c-4d31e06e1e57\") "
Sep 07 22:51:39.270913 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.270846    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:39.275102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.275068    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-140678379788592"]
Sep 07 22:51:39.276532 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.276501    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4" (OuterVolumeSpecName: "kube-api-access-cbpd4") pod "b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" (UID: "b4fc06e2-8440-4a10-8a1c-4d31e06e1e57"). InnerVolumeSpecName "kube-api-access-cbpd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:51:39.370726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.370624    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpd4\" (UniqueName: \"kubernetes.io/projected/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57-kube-api-access-cbpd4\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:51:39.475984 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.475935    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:39.476259 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.476234    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 07 22:51:39.476325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.476285    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 07 22:51:39.854021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:39.853982    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:40.090055 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.090022    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:40.090511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.090075    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="229c4db72a95fadf21b895cf38c76e7bfe9fe8a08f52167a5062c70c36235513"
Sep 07 22:51:40.100731 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:51:40.100700    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7578-140678379788592\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:40.134039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.133955    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:51:40.134482 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.134461    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:40.134571 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.134483    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:40.134637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.134574    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" containerName="test-curl-7578-140678379788592"
Sep 07 22:51:40.141819 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.141794    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.144203 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:51:40.144177    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7578-140678379788592\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:40.144409 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.144394    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-33bf198b-openshift-default-dockercfg-wrhxg\""
Sep 07 22:51:40.170149 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.165535    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.174948    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177566    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177638    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2vn\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177771    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177806    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177835    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177858    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177913    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.178165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.177939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282525 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282414    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282525 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282454    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282525 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282473    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282547    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282617    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282677    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2vn\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.282828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.283188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.282888    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.283188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.283043    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.283188 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.283117    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.283467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.283446    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.283551 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.283531    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.286166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.285120    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.286166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.285613    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.290737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.290713    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.290853 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.290838    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2vn\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn\") pod \"e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") " pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.476339 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.476301    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 07 22:51:40.476511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.476370    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 07 22:51:40.484430 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.484393    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:40.725636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.725603    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:40.766738 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.766641    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:51:40.771663 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:40.771625    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e666dc0_4c8a_4319_b04c_df3e87a25f55.slice/crio-727d5465a4d95f621d236f4027ae76f8897df1d3f0b7486a081e4baca2ca4fec WatchSource:0}: Error finding container 727d5465a4d95f621d236f4027ae76f8897df1d3f0b7486a081e4baca2ca4fec: Status 404 returned error can't find the container with id 727d5465a4d95f621d236f4027ae76f8897df1d3f0b7486a081e4baca2ca4fec
Sep 07 22:51:40.774536 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.774498    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:40.774637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.774581    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:40.774637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.774621    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:51:40.796872 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:51:40.796837    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7578-140678379788592\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" pod="opendatahub/test-curl-7578-140678379788592"
Sep 07 22:51:40.799228 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:40.799175    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fc06e2-8440-4a10-8a1c-4d31e06e1e57" path="/var/lib/kubelet/pods/b4fc06e2-8440-4a10-8a1c-4d31e06e1e57/volumes"
Sep 07 22:51:41.105274 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.105174    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerStarted","Data":"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"}
Sep 07 22:51:41.105692 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.105273    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerStarted","Data":"727d5465a4d95f621d236f4027ae76f8897df1d3f0b7486a081e4baca2ca4fec"}
Sep 07 22:51:41.125814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.125762    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podStartSLOduration=1.125747817 podStartE2EDuration="1.125747817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:41.124642086 +0000 UTC m=+1180.966910355" watchObservedRunningTime="2026-09-07 22:51:41.125747817 +0000 UTC m=+1180.968016074"
Sep 07 22:51:41.476302 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.476259    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 07 22:51:41.476481 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.476323    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 07 22:51:41.485429 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.485401    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:41.490462 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.490438    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:41.851903 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:41.851776    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:42.110834 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:42.110748    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:42.111671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:42.111649    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:51:42.475635 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:42.475592    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 07 22:51:42.475825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:42.475680    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 07 22:51:43.025050 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:43.025011    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:43.118818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:43.118782    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:43.476052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:43.476011    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 07 22:51:43.476317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:43.476089    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 07 22:51:44.235438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.235395    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:44.335424 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.335386    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:44.480281 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.480252    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:44.480581 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.480553    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:44.481389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.481370    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 22:51:44.737680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.737645    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:44.802692 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:44.802659    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:45.114039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:45.113941    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:45.792679 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:45.792639    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:50.635770 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:50.635734    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:50.688793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:50.688761    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:51.169023 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:51.168986    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:51:57.781626 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.781586    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:51:57.788986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.788957    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:51:57.789200 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.789049    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.790327 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.790293    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:51:57.791492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.791458    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"auth-test-model-1d0395-kserve-self-signed-certs\""
Sep 07 22:51:57.792594 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.792572    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"openshift-service-ca.crt\""
Sep 07 22:51:57.792742 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.792717    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"kube-root-ca.crt\""
Sep 07 22:51:57.792856 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.792675    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"default-dockercfg-cmstf\""
Sep 07 22:51:57.878721 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.878678    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.878925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.878778    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.878925 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.878813    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zd8h\" (UniqueName: \"kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.879064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.878951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.879064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.879045    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.879226 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.879072    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980018 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.979981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980034    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zd8h\" (UniqueName: \"kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980216 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980188    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980276    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980349 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980302    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980461 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980354    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980675 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980652    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.980800 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.980778    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.982361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.982337    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.982786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.982764    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:57.987859 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:57.987832    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zd8h\" (UniqueName: \"kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h\") pod \"auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:58.099459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:58.099425    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:51:58.372881 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:58.372855    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:51:58.374667 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:51:58.374639    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc862576d_6a0b_4a9b_b907_6da1a25254ec.slice/crio-abca66748d12b5fdafc1223b310851a99d1de80d4ca7337ad543eb9e32ed7ee0 WatchSource:0}: Error finding container abca66748d12b5fdafc1223b310851a99d1de80d4ca7337ad543eb9e32ed7ee0: Status 404 returned error can't find the container with id abca66748d12b5fdafc1223b310851a99d1de80d4ca7337ad543eb9e32ed7ee0
Sep 07 22:51:59.217022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:59.216973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" event={"ID":"c862576d-6a0b-4a9b-b907-6da1a25254ec","Type":"ContainerStarted","Data":"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"}
Sep 07 22:51:59.217022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:59.217029    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" event={"ID":"c862576d-6a0b-4a9b-b907-6da1a25254ec","Type":"ContainerStarted","Data":"abca66748d12b5fdafc1223b310851a99d1de80d4ca7337ad543eb9e32ed7ee0"}
Sep 07 22:51:59.233206 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:51:59.233155    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" podStartSLOduration=2.233121125 podStartE2EDuration="2.233121125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:59.231680715 +0000 UTC m=+1199.073948974" watchObservedRunningTime="2026-09-07 22:51:59.233121125 +0000 UTC m=+1199.075389391"
Sep 07 22:52:00.791337 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:00.791299    2550 scope.go:117] "RemoveContainer" containerID="340497811cb7cd96a32c26adeb806e15a94fcec4f59214cd96b20306d8daa21e"
Sep 07 22:52:01.230585 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.230554    2550 generic.go:356] "Generic (PLEG): container finished" podID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerID="3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8" exitCode=0
Sep 07 22:52:01.230726 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.230625    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerDied","Data":"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"}
Sep 07 22:52:01.231049 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.231035    2550 scope.go:117] "RemoveContainer" containerID="3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"
Sep 07 22:52:01.232241 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.232211    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:52:01.232315 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.232285    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:52:01.232361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:01.232316    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:52:02.239273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:02.239234    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerStarted","Data":"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"}
Sep 07 22:52:02.484460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:02.484411    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:52:02.484636 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:02.484499    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:52:02.484775 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:02.484743    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:02.484850 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:02.484803    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:03.485805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:03.485758    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:03.486365 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:03.485826    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:04.485355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:04.485323    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:04.485540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:04.485371    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:05.485355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:05.485317    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:05.485745 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:05.485386    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:06.485841 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:06.485802    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:06.486294 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:06.485864    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:07.485438 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:07.485399    2550 patch_prober.go:28] interesting pod/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 07 22:52:07.485617 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:07.485461    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 07 22:52:07.988105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:07.988075    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:52:08.100667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.100611    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:52:08.100872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.100680    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:52:08.120830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.120793    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:52:08.296554 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.296463    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:52:08.488846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.488813    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:52:08.489120 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.489096    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:52:08.489905 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:08.489886    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:52:14.083328 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:14.083292    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:52:21.368511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:21.368477    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:52:45.279405 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.279366    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:52:45.285440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.283329    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.288939 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.288913    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"default-dockercfg-6pjx2\""
Sep 07 22:52:45.289657 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.289635    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"openshift-service-ca.crt\""
Sep 07 22:52:45.290010 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.289989    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"auth-test-model-7823cc-kserve-self-signed-certs\""
Sep 07 22:52:45.290607 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.290578    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"kube-root-ca.crt\""
Sep 07 22:52:45.297846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.297771    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:52:45.309979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.309954    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:52:45.432902 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.432873    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.433024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.432915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.433024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.432950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.433024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.432977    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.433024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.433002    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m2bs\" (UniqueName: \"kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.433207 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.433028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534337 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534337 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534431    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534523 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534559 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m2bs\" (UniqueName: \"kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534601 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534571    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534634    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534801 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534777    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.534878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.534861    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.536631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.536612    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.536764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.536748    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.542480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.542432    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m2bs\" (UniqueName: \"kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs\") pod \"auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.601122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.601083    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:45.873357 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:45.873329    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:52:45.874843 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:52:45.874812    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc8c2538_6515_4556_8fca_48dcd4828ed4.slice/crio-2480b7118ce351a0a1b5fcb8f9a13151bbc2a2579364c0514706c67e81f1e623 WatchSource:0}: Error finding container 2480b7118ce351a0a1b5fcb8f9a13151bbc2a2579364c0514706c67e81f1e623: Status 404 returned error can't find the container with id 2480b7118ce351a0a1b5fcb8f9a13151bbc2a2579364c0514706c67e81f1e623
Sep 07 22:52:46.508340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:46.508306    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" event={"ID":"bc8c2538-6515-4556-8fca-48dcd4828ed4","Type":"ContainerStarted","Data":"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"}
Sep 07 22:52:46.508340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:46.508346    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" event={"ID":"bc8c2538-6515-4556-8fca-48dcd4828ed4","Type":"ContainerStarted","Data":"2480b7118ce351a0a1b5fcb8f9a13151bbc2a2579364c0514706c67e81f1e623"}
Sep 07 22:52:46.523949 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:46.523900    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" podStartSLOduration=1.523885157 podStartE2EDuration="1.523885157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:52:46.522624697 +0000 UTC m=+1246.364892974" watchObservedRunningTime="2026-09-07 22:52:46.523885157 +0000 UTC m=+1246.366153420"
Sep 07 22:52:53.264316 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:53.264224    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:52:54.154720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:54.154685    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:52:55.601355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:55.601324    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:55.601743 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:55.601366    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:55.613753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:55.613723    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:52:56.582646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:52:56.582614    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:53:00.832182 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:00.832147    2550 scope.go:117] "RemoveContainer" containerID="2798831447a21461cab839d0ed06fa9ae9a6c24bed91bdd5aa3017c93d3facbb"
Sep 07 22:53:11.173064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:11.173027    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:53:18.004244 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.004210    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"]
Sep 07 22:53:18.009009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.008991    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:18.011793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.011765    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:53:18.011947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.011870    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:53:18.014361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.014336    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"]
Sep 07 22:53:18.129290 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.129245    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4\" (UID: \"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:18.230566 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.230531    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4\" (UID: \"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:18.231185 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.231166    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4\" (UID: \"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:18.320539 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.320471    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:18.587623 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.587598    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"]
Sep 07 22:53:18.589223 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:53:18.589195    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eacd1d9_6dcf_4d84_bc8b_f4c3248255ae.slice/crio-2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21 WatchSource:0}: Error finding container 2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21: Status 404 returned error can't find the container with id 2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21
Sep 07 22:53:18.708516 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:18.708479    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4" event={"ID":"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae","Type":"ContainerStarted","Data":"2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21"}
Sep 07 22:53:19.716426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:19.716388    2550 generic.go:356] "Generic (PLEG): container finished" podID="9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" containerID="b91a30a6fa997b4abce4727655eff3257656d3c71bbd2281c75abac3d7c07f8b" exitCode=0
Sep 07 22:53:19.716789 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:19.716537    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4" event={"ID":"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae","Type":"ContainerDied","Data":"b91a30a6fa997b4abce4727655eff3257656d3c71bbd2281c75abac3d7c07f8b"}
Sep 07 22:53:20.857201 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:20.857178    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:20.955403 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:20.955377    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca\") pod \"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae\" (UID: \"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae\") "
Sep 07 22:53:20.955735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:20.955716    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" (UID: "9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:53:21.056159 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:21.056079    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:53:21.731604 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:21.731571    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"
Sep 07 22:53:21.731782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:21.731614    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4" event={"ID":"9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae","Type":"ContainerDied","Data":"2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21"}
Sep 07 22:53:21.731782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:21.731667    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2502ae4194e2c2840a98806ec1f7cb44d53d86906fc60d5ee5b351a731b64f21"
Sep 07 22:53:33.522852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.522819    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:33.657574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.657538    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:33.754515 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.754479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:33.790975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.790891    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"]
Sep 07 22:53:33.793070 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.793049    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-33bf198b-g6lj4"]
Sep 07 22:53:33.813447 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:33.813418    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:34.795968 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:34.795933    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" path="/var/lib/kubelet/pods/9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae/volumes"
Sep 07 22:53:36.369199 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:36.369161    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:53:36.768965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:36.768936    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:41.844559 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:41.844523    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:53:51.767009 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:53:51.766973    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:54:00.855831 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:00.855803    2550 scope.go:117] "RemoveContainer" containerID="1d47e52220fb588ba71e93e0581f7ebd3448d222559269b7f67cf34b86f72b99"
Sep 07 22:54:00.864312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:00.864287    2550 scope.go:117] "RemoveContainer" containerID="ae15a44e1dda1dfad9c8bbd34ac2b6254b9cd13d3b6be4014d365ee185a1c8bc"
Sep 07 22:54:07.965156 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:07.965103    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:54:18.720224 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:18.720183    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:18.887809 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:18.887772    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:24.295652 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:24.295566    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:24.363862 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:24.363827    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:54:40.361835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:40.361797    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:54:55.323213 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:55.323154    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:55.422564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:55.422531    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:55.540760 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:55.540724    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:54:55.541042 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:55.541011    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy" containerID="cri-o://479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca" gracePeriod=30
Sep 07 22:54:55.555061 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:55.555027    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:56.731849 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:56.731817    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:56.831589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:56.831548    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:56.931570 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:56.931527    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:57.375975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:57.375932    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:54:57.523489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:57.523460    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:57.623837 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:57.623779    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:57.724109 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:57.724078    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:54:58.951727 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:54:58.951691    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:00.795489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.795465    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:55:00.949078 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949048    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949096    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949157    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949266 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf2vn\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949272    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949336    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949368    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949393    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949459 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949427    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data\") pod \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\" (UID: \"5e666dc0-4c8a-4319-b04c-df3e87a25f55\") "
Sep 07 22:55:00.949731 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949624    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:00.949795 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949739    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:55:00.949937 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949846    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:00.950006 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949948    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:00.950006 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.949972    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:00.950121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.950068    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:00.950352 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.950323    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:00.951510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.951481    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:55:00.951510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.951489    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:00.951665 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.951494    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:55:00.951665 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:00.951567    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn" (OuterVolumeSpecName: "kube-api-access-kf2vn") pod "5e666dc0-4c8a-4319-b04c-df3e87a25f55" (UID: "5e666dc0-4c8a-4319-b04c-df3e87a25f55"). InnerVolumeSpecName "kube-api-access-kf2vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:55:01.050630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050581    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050624    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050630 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050638    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050651    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050661    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050671    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e666dc0-4c8a-4319-b04c-df3e87a25f55-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.050877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.050681    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2vn\" (UniqueName: \"kubernetes.io/projected/5e666dc0-4c8a-4319-b04c-df3e87a25f55-kube-api-access-kf2vn\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:01.320285 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.320181    2550 generic.go:356] "Generic (PLEG): container finished" podID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerID="479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca" exitCode=0
Sep 07 22:55:01.320285 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.320241    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"
Sep 07 22:55:01.320285 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.320259    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerDied","Data":"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"}
Sep 07 22:55:01.320561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.320336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd" event={"ID":"5e666dc0-4c8a-4319-b04c-df3e87a25f55","Type":"ContainerDied","Data":"727d5465a4d95f621d236f4027ae76f8897df1d3f0b7486a081e4baca2ca4fec"}
Sep 07 22:55:01.320561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.320362    2550 scope.go:117] "RemoveContainer" containerID="479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"
Sep 07 22:55:01.330854 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.330834    2550 scope.go:117] "RemoveContainer" containerID="3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"
Sep 07 22:55:01.339122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.339104    2550 scope.go:117] "RemoveContainer" containerID="479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"
Sep 07 22:55:01.339465 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:55:01.339435    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca\": container with ID starting with 479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca not found: ID does not exist" containerID="479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"
Sep 07 22:55:01.339558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.339475    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca"} err="failed to get container status \"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca\": rpc error: code = NotFound desc = could not find container \"479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca\": container with ID starting with 479c074554844d04a0a7f10fa87243a17d0c05c8f950aa4d3798bd3393f450ca not found: ID does not exist"
Sep 07 22:55:01.339558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.339494    2550 scope.go:117] "RemoveContainer" containerID="3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"
Sep 07 22:55:01.339829 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:55:01.339796    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8\": container with ID starting with 3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8 not found: ID does not exist" containerID="3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"
Sep 07 22:55:01.339977 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.339836    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8"} err="failed to get container status \"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8\": rpc error: code = NotFound desc = could not find container \"3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8\": container with ID starting with 3c8a5e82217620a606e95287aa96b730899d78878265fbe3176e3d90f975dea8 not found: ID does not exist"
Sep 07 22:55:01.343181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.343160    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:55:01.347810 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:01.347788    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-33bf198b-openshift-default-cfdcb9956-4mvfd"]
Sep 07 22:55:02.796532 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:02.796496    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" path="/var/lib/kubelet/pods/5e666dc0-4c8a-4319-b04c-df3e87a25f55/volumes"
Sep 07 22:55:04.659420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:04.659385    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:04.717962 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:04.717928    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:05.295960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:05.295920    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:07.658728 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:07.658688    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:11.086510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:11.086479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:11.146102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:11.146075    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:11.224792 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:11.224742    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:11.321467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:11.321436    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:12.565164 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:12.565115    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:55:14.125730 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:14.125689    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:55:14.149171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:14.149118    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:15.150917 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:15.150882    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:55:15.174122 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:15.174092    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:21.761057 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:21.761024    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:21.969075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:21.969045    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:23.491511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:23.491475    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:26.865895 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:26.865864    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:55:29.146082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.146023    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" podUID="c862576d-6a0b-4a9b-b907-6da1a25254ec" containerName="main" containerID="cri-o://00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1" gracePeriod=45
Sep 07 22:55:29.398939 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.398885    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:55:29.492044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492012    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zd8h\" (UniqueName: \"kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492061    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492120    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492189    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492236 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492234    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492463 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492285    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home\") pod \"c862576d-6a0b-4a9b-b907-6da1a25254ec\" (UID: \"c862576d-6a0b-4a9b-b907-6da1a25254ec\") "
Sep 07 22:55:29.492463 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492330    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:29.492570 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492468    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:29.492706 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home" (OuterVolumeSpecName: "home") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:29.492836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492686    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:29.492836 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.492759    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:29.494683 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.494657    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm" (OuterVolumeSpecName: "dshm") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:29.494902 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.494882    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h" (OuterVolumeSpecName: "kube-api-access-9zd8h") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "kube-api-access-9zd8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:55:29.495053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495020    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c862576d-6a0b-4a9b-b907-6da1a25254ec" (UID: "c862576d-6a0b-4a9b-b907-6da1a25254ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:55:29.495846 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495818    2550 generic.go:356] "Generic (PLEG): container finished" podID="c862576d-6a0b-4a9b-b907-6da1a25254ec" containerID="00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1" exitCode=0
Sep 07 22:55:29.495956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495869    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"
Sep 07 22:55:29.495956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495904    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" event={"ID":"c862576d-6a0b-4a9b-b907-6da1a25254ec","Type":"ContainerDied","Data":"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"}
Sep 07 22:55:29.496083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495958    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj" event={"ID":"c862576d-6a0b-4a9b-b907-6da1a25254ec","Type":"ContainerDied","Data":"abca66748d12b5fdafc1223b310851a99d1de80d4ca7337ad543eb9e32ed7ee0"}
Sep 07 22:55:29.496083 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.495982    2550 scope.go:117] "RemoveContainer" containerID="00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"
Sep 07 22:55:29.515021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.514998    2550 scope.go:117] "RemoveContainer" containerID="00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"
Sep 07 22:55:29.515334 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:55:29.515315    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1\": container with ID starting with 00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1 not found: ID does not exist" containerID="00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"
Sep 07 22:55:29.515431 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.515342    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1"} err="failed to get container status \"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1\": rpc error: code = NotFound desc = could not find container \"00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1\": container with ID starting with 00438788d888dbb26569d2c4c3579733c25f5af2caddec63b4dc6f40c25083a1 not found: ID does not exist"
Sep 07 22:55:29.525668 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.525641    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:55:29.530565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.530541    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/auth-test-model-1d0395-kserve-67f5f8c5c9-x7wnj"]
Sep 07 22:55:29.593783 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.593759    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862576d-6a0b-4a9b-b907-6da1a25254ec-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:29.593783 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.593780    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:29.593922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.593788    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862576d-6a0b-4a9b-b907-6da1a25254ec-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:29.593922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:29.593797    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zd8h\" (UniqueName: \"kubernetes.io/projected/c862576d-6a0b-4a9b-b907-6da1a25254ec-kube-api-access-9zd8h\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.174008 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.173970    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" podUID="bc8c2538-6515-4556-8fca-48dcd4828ed4" containerName="main" containerID="cri-o://8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d" gracePeriod=45
Sep 07 22:55:30.424454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.424400    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:55:30.500383 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500358    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500383 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500387    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500424    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500472    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m2bs\" (UniqueName: \"kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500531    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500552    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home\") pod \"bc8c2538-6515-4556-8fca-48dcd4828ed4\" (UID: \"bc8c2538-6515-4556-8fca-48dcd4828ed4\") "
Sep 07 22:55:30.500812 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500594    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:30.500812 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500698    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:30.500919 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500846    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home" (OuterVolumeSpecName: "home") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:30.500975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500924    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.500975 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.500943    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.502828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.502803    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm" (OuterVolumeSpecName: "dshm") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:55:30.503082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.503047    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:55:30.503082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.503064    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs" (OuterVolumeSpecName: "kube-api-access-9m2bs") pod "bc8c2538-6515-4556-8fca-48dcd4828ed4" (UID: "bc8c2538-6515-4556-8fca-48dcd4828ed4"). InnerVolumeSpecName "kube-api-access-9m2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:55:30.504389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.504363    2550 generic.go:356] "Generic (PLEG): container finished" podID="bc8c2538-6515-4556-8fca-48dcd4828ed4" containerID="8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d" exitCode=0
Sep 07 22:55:30.504477 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.504410    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"
Sep 07 22:55:30.504477 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.504440    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" event={"ID":"bc8c2538-6515-4556-8fca-48dcd4828ed4","Type":"ContainerDied","Data":"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"}
Sep 07 22:55:30.504613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.504480    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5" event={"ID":"bc8c2538-6515-4556-8fca-48dcd4828ed4","Type":"ContainerDied","Data":"2480b7118ce351a0a1b5fcb8f9a13151bbc2a2579364c0514706c67e81f1e623"}
Sep 07 22:55:30.504613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.504498    2550 scope.go:117] "RemoveContainer" containerID="8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"
Sep 07 22:55:30.521921 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.521903    2550 scope.go:117] "RemoveContainer" containerID="8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"
Sep 07 22:55:30.522201 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:55:30.522185    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d\": container with ID starting with 8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d not found: ID does not exist" containerID="8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"
Sep 07 22:55:30.522265 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.522208    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d"} err="failed to get container status \"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d\": rpc error: code = NotFound desc = could not find container \"8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d\": container with ID starting with 8c84882ffc6e5c7dd2f16aee2227736d56dd915b5c0daff68909029bf9df254d not found: ID does not exist"
Sep 07 22:55:30.531741 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.531716    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:55:30.534834 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.534811    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/auth-test-model-7823cc-kserve-58bbf4cd67-qw2d5"]
Sep 07 22:55:30.601376 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.601343    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.601376 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.601371    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m2bs\" (UniqueName: \"kubernetes.io/projected/bc8c2538-6515-4556-8fca-48dcd4828ed4-kube-api-access-9m2bs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.601549 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.601382    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8c2538-6515-4556-8fca-48dcd4828ed4-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.601549 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.601391    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8c2538-6515-4556-8fca-48dcd4828ed4-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:55:30.796427 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.796401    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8c2538-6515-4556-8fca-48dcd4828ed4" path="/var/lib/kubelet/pods/bc8c2538-6515-4556-8fca-48dcd4828ed4/volumes"
Sep 07 22:55:30.796826 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:30.796812    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c862576d-6a0b-4a9b-b907-6da1a25254ec" path="/var/lib/kubelet/pods/c862576d-6a0b-4a9b-b907-6da1a25254ec/volumes"
Sep 07 22:55:42.862865 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:42.862833    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:45.772906 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.772871    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773321    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8c2538-6515-4556-8fca-48dcd4828ed4" containerName="main"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773334    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8c2538-6515-4556-8fca-48dcd4828ed4" containerName="main"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773346    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773351    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773358    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773364    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773371    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" containerName="revoke-keys"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773376    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" containerName="revoke-keys"
Sep 07 22:55:45.773380 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773383    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c862576d-6a0b-4a9b-b907-6da1a25254ec" containerName="main"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773388    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c862576d-6a0b-4a9b-b907-6da1a25254ec" containerName="main"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773458    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c862576d-6a0b-4a9b-b907-6da1a25254ec" containerName="main"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773465    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773469    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eacd1d9-6dcf-4d84-bc8b-f4c3248255ae" containerName="revoke-keys"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773478    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e666dc0-4c8a-4319-b04c-df3e87a25f55" containerName="istio-proxy"
Sep 07 22:55:45.773680 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.773485    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8c2538-6515-4556-8fca-48dcd4828ed4" containerName="main"
Sep 07 22:55:45.778030 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.778006    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.780798 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.780769    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"openshift-service-ca.crt\""
Sep 07 22:55:45.780798 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.780795    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"kube-root-ca.crt\""
Sep 07 22:55:45.781990 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.781970    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"sub-test-model-7823cc-kserve-self-signed-certs\""
Sep 07 22:55:45.782118 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.782007    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"default-dockercfg-6pjx2\""
Sep 07 22:55:45.785820 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.785515    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:55:45.795610 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.795573    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:45.825270 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825240    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8ng\" (UniqueName: \"kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.825378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825282    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.825378 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825343    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.825472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825375    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.825472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.825540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.825513    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926465 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926437    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926487    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926536    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8ng\" (UniqueName: \"kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926679 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926567    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926688    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926735 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926724    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926897 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926880    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.926998 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.926982    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.927088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.927069    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.928685 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.928663    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.929021 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.929004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:45.933973 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:45.933950    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8ng\" (UniqueName: \"kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng\") pod \"sub-test-model-7823cc-kserve-5656797867-fwlvz\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:46.088466 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:46.088383    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:46.359346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:46.359318    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:55:46.360759 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:55:46.360729    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a2e723_b5c9_4d66_9249_be1844318063.slice/crio-7ffe76caab33c3671dc7d1549f363734c3b7e15039777e665a03ce0cd391ec8e WatchSource:0}: Error finding container 7ffe76caab33c3671dc7d1549f363734c3b7e15039777e665a03ce0cd391ec8e: Status 404 returned error can't find the container with id 7ffe76caab33c3671dc7d1549f363734c3b7e15039777e665a03ce0cd391ec8e
Sep 07 22:55:46.598838 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:46.598797    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" event={"ID":"c8a2e723-b5c9-4d66-9249-be1844318063","Type":"ContainerStarted","Data":"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"}
Sep 07 22:55:46.598997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:46.598854    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" event={"ID":"c8a2e723-b5c9-4d66-9249-be1844318063","Type":"ContainerStarted","Data":"7ffe76caab33c3671dc7d1549f363734c3b7e15039777e665a03ce0cd391ec8e"}
Sep 07 22:55:46.614855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:46.614770    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" podStartSLOduration=1.614755196 podStartE2EDuration="1.614755196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:46.613929207 +0000 UTC m=+1426.456197474" watchObservedRunningTime="2026-09-07 22:55:46.614755196 +0000 UTC m=+1426.457023454"
Sep 07 22:55:48.721701 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:48.721669    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:48.787330 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:48.787286    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:49.155840 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.155730    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:49.349759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.349729    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:55:49.353529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.353499    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.356269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.356244    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dd5b43-openshift-default-dockercfg-zgczq\""
Sep 07 22:55:49.366058 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.366030    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:55:49.376493 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.376466    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:49.457670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457630    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.457873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.457873 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.457993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457871    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.457993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457917    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.457993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.458177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.457996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.458177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.458069    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8btr\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.458177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.458112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559094 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8btr\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559110    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559193    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559269 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559216    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559294    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559320    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559420 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559352    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559506    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559573 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559542    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559682 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559589    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559682 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559625    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559789 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559714    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.559937 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.559911    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.560172 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.560155    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.562110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.562090    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.562372 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.562352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.566747 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.566728    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.566870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.566791    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8btr\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr\") pod \"e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") " pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.666727 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.666705    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:49.691965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.691931    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:49.878263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.878216    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:55:49.947435 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.947408    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:55:49.949995 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:55:49.949962    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc43396_bfe7_4cee_8ab8_edb244d026c6.slice/crio-2e3473faf0dbd9a503a4da3258fc785c0a641bb8317c97020599388a097d71f2 WatchSource:0}: Error finding container 2e3473faf0dbd9a503a4da3258fc785c0a641bb8317c97020599388a097d71f2: Status 404 returned error can't find the container with id 2e3473faf0dbd9a503a4da3258fc785c0a641bb8317c97020599388a097d71f2
Sep 07 22:55:49.952388 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.952358    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:55:49.952472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.952419    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:55:49.952472 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:49.952448    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:55:50.622148 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:50.622092    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerStarted","Data":"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"}
Sep 07 22:55:50.622340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:50.622169    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerStarted","Data":"2e3473faf0dbd9a503a4da3258fc785c0a641bb8317c97020599388a097d71f2"}
Sep 07 22:55:50.642522 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:50.642472    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podStartSLOduration=1.642455963 podStartE2EDuration="1.642455963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:50.640328143 +0000 UTC m=+1430.482596412" watchObservedRunningTime="2026-09-07 22:55:50.642455963 +0000 UTC m=+1430.484724230"
Sep 07 22:55:50.667533 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:50.667507    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:51.672311 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:51.672284    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:52.542934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:52.542895    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:52.635204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:52.635171    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:52.636138 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:52.636102    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:55:52.640461 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:52.640437    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:53.699993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:53.699959    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:53.797809 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:53.797776    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:54.047529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:54.047445    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:54.164694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:54.164662    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:54.258275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:54.258245    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:55.503434 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:55.503390    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:56.089052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:56.089011    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:56.089261 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:56.089067    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:56.101981 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:56.101956    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:55:56.515044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:56.515011    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:55:56.669950 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:55:56.669915    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:56:00.898020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:00.897985    2550 scope.go:117] "RemoveContainer" containerID="9725cc0ec045bad44ae32312e6071d8b8b300d041a31efe35dfb2aaade7046fe"
Sep 07 22:56:00.906637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:00.906617    2550 scope.go:117] "RemoveContainer" containerID="fe14f6f6f0a4b7fc3ef79618884d08511c35bbd8c52bdded0fbf5d9062c891f2"
Sep 07 22:56:08.677263 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:08.677216    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:56:08.677637 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:08.677295    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:56:09.067670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:09.067636    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:56:10.743256 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.743222    2550 generic.go:356] "Generic (PLEG): container finished" podID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerID="73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471" exitCode=0
Sep 07 22:56:10.743678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.743305    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerDied","Data":"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"}
Sep 07 22:56:10.743737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.743693    2550 scope.go:117] "RemoveContainer" containerID="73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"
Sep 07 22:56:10.744877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.744850    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:10.744945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.744918    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:10.744987 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:10.744951    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:11.779710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:11.779658    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerStarted","Data":"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"}
Sep 07 22:56:12.667437 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:12.667397    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:56:12.667642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:12.667541    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 07 22:56:12.667642 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:12.667623    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 07 22:56:13.668157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:13.668095    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 07 22:56:13.668782 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:13.668744    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 07 22:56:14.668589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:14.668096    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 07 22:56:14.668589 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:14.668197    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 07 22:56:15.382717 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:15.382681    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:15.668043 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:15.667943    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 07 22:56:15.668043 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:15.668023    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 07 22:56:16.116234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:16.116194    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:16.667475 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:16.667423    2550 patch_prober.go:28] interesting pod/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 07 22:56:16.667807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:16.667499    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 07 22:56:17.478678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:17.478644    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:17.671502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:17.671471    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:56:17.671789 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:17.671766    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:56:17.672543 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:17.672526    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:56:17.896534 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:17.896456    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:18.161855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.161775    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:56:18.166626 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.166603    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.169230 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.169175    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c7776f-openshift-default-dockercfg-44r4h\""
Sep 07 22:56:18.174991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.174966    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:56:18.191085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.191048    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:18.229352 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229325    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229364    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229451    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229514    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229605 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229563    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229709 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229620    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229709 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229655    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229777 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229712    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnm5b\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.229777 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.229746    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.294179 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.294122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:18.330299 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330267    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330315    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330439 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330400    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330591    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330648 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330628    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330641    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330658    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnm5b\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330762 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330749    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.330960 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330925    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.331044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.330958    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.331463 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.331438    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.332833 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.332815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.332994 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.332974    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.337759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.337738    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.337924 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.337904    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnm5b\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b\") pod \"e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") " pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.480325 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.480289    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:18.757398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.757365    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:56:18.758120 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:56:18.758086    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c36434_b9b8_4e53_9bc0_b62c5bd6175e.slice/crio-cb799d2c59adb3e783925eb5d025b8da51b4e787a89e7c0787a1964586d849f1 WatchSource:0}: Error finding container cb799d2c59adb3e783925eb5d025b8da51b4e787a89e7c0787a1964586d849f1: Status 404 returned error can't find the container with id cb799d2c59adb3e783925eb5d025b8da51b4e787a89e7c0787a1964586d849f1
Sep 07 22:56:18.760444 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.760414    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:18.760516 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.760480    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:18.760516 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.760505    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:18.825806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:18.825777    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerStarted","Data":"cb799d2c59adb3e783925eb5d025b8da51b4e787a89e7c0787a1964586d849f1"}
Sep 07 22:56:19.833275 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:19.833244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerStarted","Data":"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"}
Sep 07 22:56:19.855316 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:19.855274    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podStartSLOduration=1.8552588829999999 podStartE2EDuration="1.855258883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:56:19.85304851 +0000 UTC m=+1459.695316778" watchObservedRunningTime="2026-09-07 22:56:19.855258883 +0000 UTC m=+1459.697527215"
Sep 07 22:56:20.481483 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:20.481449    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:20.486722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:20.486697    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:20.838480 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:20.838403    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:20.839368 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:20.839348    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:21.723071 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:21.723025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:21.816441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:21.816408    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:22.983470 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:22.983442    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:23.084027 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:23.083993    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:23.184413 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:23.184381    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:23.439076 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:23.439049    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:23.553823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:23.553792    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:23.911786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:23.911700    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:24.569568 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:24.569536    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:56:24.626756 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:24.626721    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:26.615321 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.615284    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7581-73ff75"]
Sep 07 22:56:26.618563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.618544    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:26.621281 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.621255    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b689c\""
Sep 07 22:56:26.624759 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.624732    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-73ff75"]
Sep 07 22:56:26.713398 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.713363    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g624p\" (UniqueName: \"kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p\") pod \"mt-curl-7581-73ff75\" (UID: \"bd9d9a00-c53a-4a48-a4ae-c9b09c796546\") " pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:26.814467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.814434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g624p\" (UniqueName: \"kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p\") pod \"mt-curl-7581-73ff75\" (UID: \"bd9d9a00-c53a-4a48-a4ae-c9b09c796546\") " pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:26.822087 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.822055    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g624p\" (UniqueName: \"kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p\") pod \"mt-curl-7581-73ff75\" (UID: \"bd9d9a00-c53a-4a48-a4ae-c9b09c796546\") " pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:26.929740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:26.929667    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:27.199178 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:27.199151    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-73ff75"]
Sep 07 22:56:27.200707 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:56:27.200680    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd9d9a00_c53a_4a48_a4ae_c9b09c796546.slice/crio-0fd1743f125a4673ab139d95f6eee03dff8185a365f0e584103f079a01f7a945 WatchSource:0}: Error finding container 0fd1743f125a4673ab139d95f6eee03dff8185a365f0e584103f079a01f7a945: Status 404 returned error can't find the container with id 0fd1743f125a4673ab139d95f6eee03dff8185a365f0e584103f079a01f7a945
Sep 07 22:56:27.202075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:27.202052    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:56:27.881267 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:27.881239    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-73ff75" event={"ID":"bd9d9a00-c53a-4a48-a4ae-c9b09c796546","Type":"ContainerStarted","Data":"0fd1743f125a4673ab139d95f6eee03dff8185a365f0e584103f079a01f7a945"}
Sep 07 22:56:28.887945 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:28.887908    2550 generic.go:356] "Generic (PLEG): container finished" podID="bd9d9a00-c53a-4a48-a4ae-c9b09c796546" containerID="ce8f67cd4bc6822b9cc79c57a24231857d568e146f68a83ad6f316bcc8e170c3" exitCode=0
Sep 07 22:56:28.888385 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:28.888002    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-73ff75" event={"ID":"bd9d9a00-c53a-4a48-a4ae-c9b09c796546","Type":"ContainerDied","Data":"ce8f67cd4bc6822b9cc79c57a24231857d568e146f68a83ad6f316bcc8e170c3"}
Sep 07 22:56:30.028876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.028854    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:30.047274 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.047247    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7581-73ff75_bd9d9a00-c53a-4a48-a4ae-c9b09c796546/mt-curl-7581-73ff75/0.log"
Sep 07 22:56:30.069992 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.069967    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7581-73ff75"]
Sep 07 22:56:30.073753 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.073731    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7581-73ff75"]
Sep 07 22:56:30.144818 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.144759    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g624p\" (UniqueName: \"kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p\") pod \"bd9d9a00-c53a-4a48-a4ae-c9b09c796546\" (UID: \"bd9d9a00-c53a-4a48-a4ae-c9b09c796546\") "
Sep 07 22:56:30.146834 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.146808    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p" (OuterVolumeSpecName: "kube-api-access-g624p") pod "bd9d9a00-c53a-4a48-a4ae-c9b09c796546" (UID: "bd9d9a00-c53a-4a48-a4ae-c9b09c796546"). InnerVolumeSpecName "kube-api-access-g624p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:56:30.186904 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.186876    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7581-0f4cdf"]
Sep 07 22:56:30.187511 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.187495    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9d9a00-c53a-4a48-a4ae-c9b09c796546" containerName="mt-curl-7581-73ff75"
Sep 07 22:56:30.187560 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.187514    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9d9a00-c53a-4a48-a4ae-c9b09c796546" containerName="mt-curl-7581-73ff75"
Sep 07 22:56:30.187638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.187626    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9d9a00-c53a-4a48-a4ae-c9b09c796546" containerName="mt-curl-7581-73ff75"
Sep 07 22:56:30.191317 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.191291    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:30.192737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.192713    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-0f4cdf"]
Sep 07 22:56:30.246591 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.246560    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjr8l\" (UniqueName: \"kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l\") pod \"mt-curl-7581-0f4cdf\" (UID: \"182506de-d9fe-4f01-b324-aa7f2730095b\") " pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:30.246719 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.246628    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g624p\" (UniqueName: \"kubernetes.io/projected/bd9d9a00-c53a-4a48-a4ae-c9b09c796546-kube-api-access-g624p\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:30.348053 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.348025    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjr8l\" (UniqueName: \"kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l\") pod \"mt-curl-7581-0f4cdf\" (UID: \"182506de-d9fe-4f01-b324-aa7f2730095b\") " pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:30.355435 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.355412    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjr8l\" (UniqueName: \"kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l\") pod \"mt-curl-7581-0f4cdf\" (UID: \"182506de-d9fe-4f01-b324-aa7f2730095b\") " pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:30.502251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.502217    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:30.776100 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.776073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7581-0f4cdf"]
Sep 07 22:56:30.777946 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:56:30.777919    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182506de_d9fe_4f01_b324_aa7f2730095b.slice/crio-66c781a362435c33067391cabd44ca12efa0f8ab8f0a158d81b880f267f69f9d WatchSource:0}: Error finding container 66c781a362435c33067391cabd44ca12efa0f8ab8f0a158d81b880f267f69f9d: Status 404 returned error can't find the container with id 66c781a362435c33067391cabd44ca12efa0f8ab8f0a158d81b880f267f69f9d
Sep 07 22:56:30.797340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.797318    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd9d9a00-c53a-4a48-a4ae-c9b09c796546" path="/var/lib/kubelet/pods/bd9d9a00-c53a-4a48-a4ae-c9b09c796546/volumes"
Sep 07 22:56:30.899724 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.899692    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-0f4cdf" event={"ID":"182506de-d9fe-4f01-b324-aa7f2730095b","Type":"ContainerStarted","Data":"66c781a362435c33067391cabd44ca12efa0f8ab8f0a158d81b880f267f69f9d"}
Sep 07 22:56:30.900876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.900857    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-73ff75"
Sep 07 22:56:30.900973 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:30.900865    2550 scope.go:117] "RemoveContainer" containerID="ce8f67cd4bc6822b9cc79c57a24231857d568e146f68a83ad6f316bcc8e170c3"
Sep 07 22:56:31.909658 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:31.909622    2550 generic.go:356] "Generic (PLEG): container finished" podID="182506de-d9fe-4f01-b324-aa7f2730095b" containerID="cede1f773fa867dc481f2dd3cb652b02f217cedac1ed30aa58db4079cee20d03" exitCode=0
Sep 07 22:56:31.910022 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:31.909675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7581-0f4cdf" event={"ID":"182506de-d9fe-4f01-b324-aa7f2730095b","Type":"ContainerDied","Data":"cede1f773fa867dc481f2dd3cb652b02f217cedac1ed30aa58db4079cee20d03"}
Sep 07 22:56:33.055860 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.055836    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:33.075091 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.075055    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7581-0f4cdf_182506de-d9fe-4f01-b324-aa7f2730095b/mt-curl-7581-0f4cdf/0.log"
Sep 07 22:56:33.095866 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.095835    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7581-0f4cdf"]
Sep 07 22:56:33.099588 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.099565    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7581-0f4cdf"]
Sep 07 22:56:33.171574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.171523    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjr8l\" (UniqueName: \"kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l\") pod \"182506de-d9fe-4f01-b324-aa7f2730095b\" (UID: \"182506de-d9fe-4f01-b324-aa7f2730095b\") "
Sep 07 22:56:33.173498 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.173467    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l" (OuterVolumeSpecName: "kube-api-access-bjr8l") pod "182506de-d9fe-4f01-b324-aa7f2730095b" (UID: "182506de-d9fe-4f01-b324-aa7f2730095b"). InnerVolumeSpecName "kube-api-access-bjr8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:56:33.272121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.272092    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjr8l\" (UniqueName: \"kubernetes.io/projected/182506de-d9fe-4f01-b324-aa7f2730095b-kube-api-access-bjr8l\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:33.921998 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.921971    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c781a362435c33067391cabd44ca12efa0f8ab8f0a158d81b880f267f69f9d"
Sep 07 22:56:33.921998 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:33.921995    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:33.931943 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:56:33.931912    2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7581-0f4cdf\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="182506de-d9fe-4f01-b324-aa7f2730095b" pod="opendatahub/mt-curl-7581-0f4cdf"
Sep 07 22:56:34.796268 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:34.796238    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182506de-d9fe-4f01-b324-aa7f2730095b" path="/var/lib/kubelet/pods/182506de-d9fe-4f01-b324-aa7f2730095b/volumes"
Sep 07 22:56:35.076737 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:35.076653    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:36.091955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:36.091919    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:56:36.108488 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:36.108461    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:41.769611 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.769576    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:56:41.867261 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.867222    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:56:41.867771 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.867741    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182506de-d9fe-4f01-b324-aa7f2730095b" containerName="mt-curl-7581-0f4cdf"
Sep 07 22:56:41.867771 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.867763    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="182506de-d9fe-4f01-b324-aa7f2730095b" containerName="mt-curl-7581-0f4cdf"
Sep 07 22:56:41.867937 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.867897    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="182506de-d9fe-4f01-b324-aa7f2730095b" containerName="mt-curl-7581-0f4cdf"
Sep 07 22:56:41.871286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.871261    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.873878 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.873857    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"openshift-service-ca.crt\""
Sep 07 22:56:41.873996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.873877    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"kube-root-ca.crt\""
Sep 07 22:56:41.873996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.873896    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"rate-test-model-1d0395-kserve-self-signed-certs\""
Sep 07 22:56:41.873996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.873971    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"default-dockercfg-cmstf\""
Sep 07 22:56:41.879793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.879764    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:56:41.890404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.890371    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:41.968825 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.968793    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.968944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.968843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.968944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.968884    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.968944 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.968931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.969063 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.968997    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:41.969099 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:41.969066    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.069835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.069835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.069835 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070101 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.069946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070319 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.070298    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070364 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.070335    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.070454 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.070432    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.072874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.072850    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.073155 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.073114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.080823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.080798    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l\") pod \"rate-test-model-1d0395-kserve-645bd9db7f-l6gq7\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.185590 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.185541    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:42.485567 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.478219    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:56:42.485567 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:56:42.480853    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c405a8a_f0e4_4a57_bc04_4023ad5a43ed.slice/crio-a7248b3ab289d066a8eec2e5b965329a046b98708597b6dc38e98c764b3a8923 WatchSource:0}: Error finding container a7248b3ab289d066a8eec2e5b965329a046b98708597b6dc38e98c764b3a8923: Status 404 returned error can't find the container with id a7248b3ab289d066a8eec2e5b965329a046b98708597b6dc38e98c764b3a8923
Sep 07 22:56:42.528528 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.528495    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:42.994652 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.994608    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" event={"ID":"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed","Type":"ContainerStarted","Data":"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"}
Sep 07 22:56:42.995052 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:42.994667    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" event={"ID":"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed","Type":"ContainerStarted","Data":"a7248b3ab289d066a8eec2e5b965329a046b98708597b6dc38e98c764b3a8923"}
Sep 07 22:56:43.010254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:43.010187    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" podStartSLOduration=2.010167498 podStartE2EDuration="2.010167498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:56:43.009581439 +0000 UTC m=+1482.851849707" watchObservedRunningTime="2026-09-07 22:56:43.010167498 +0000 UTC m=+1482.852435767"
Sep 07 22:56:44.009039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:44.008998    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:44.849286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:44.849244    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:45.800615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:45.800577    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:46.013491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.013458    2550 generic.go:356] "Generic (PLEG): container finished" podID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerID="53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1" exitCode=0
Sep 07 22:56:46.013684 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.013534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerDied","Data":"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"}
Sep 07 22:56:46.013996 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.013977    2550 scope.go:117] "RemoveContainer" containerID="53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"
Sep 07 22:56:46.015181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.015154    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:46.015251 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.015219    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:46.015292 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.015252    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:56:46.655214 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:46.655183    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:47.021966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:47.021930    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerStarted","Data":"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"}
Sep 07 22:56:47.480955 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:47.480916    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:47.482274 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:47.482242    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:47.482408 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:47.482295    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:48.481540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:48.481492    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:48.482065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:48.481563    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:49.479297 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.479259    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"]
Sep 07 22:56:49.480870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.480848    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:49.480938 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.480899    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:49.482986 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.482970    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:49.485372 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.485352    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:56:49.485608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.485586    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:56:49.494330 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.494305    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"]
Sep 07 22:56:49.611346 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.611312    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:56:49.641631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.641600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k\" (UID: \"b25e1c06-7a83-41ed-ac61-096e603c6cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:49.742941 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.742853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k\" (UID: \"b25e1c06-7a83-41ed-ac61-096e603c6cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:49.743542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.743521    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k\" (UID: \"b25e1c06-7a83-41ed-ac61-096e603c6cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:49.794479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:49.794449    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:50.075191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:50.075165    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"]
Sep 07 22:56:50.076470 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:56:50.076437    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e1c06_7a83_41ed_ac61_096e603c6cdf.slice/crio-d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64 WatchSource:0}: Error finding container d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64: Status 404 returned error can't find the container with id d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64
Sep 07 22:56:50.481488 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:50.481464    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:50.481579 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:50.481505    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:50.839166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:50.839070    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:51.046709 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.046325    2550 generic.go:356] "Generic (PLEG): container finished" podID="b25e1c06-7a83-41ed-ac61-096e603c6cdf" containerID="f9e922cc92b1ff7add1019150d96cfeb93b48bfe8c09b01780369c5c6ac7e34a" exitCode=0
Sep 07 22:56:51.047075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.046418    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k" event={"ID":"b25e1c06-7a83-41ed-ac61-096e603c6cdf","Type":"ContainerDied","Data":"f9e922cc92b1ff7add1019150d96cfeb93b48bfe8c09b01780369c5c6ac7e34a"}
Sep 07 22:56:51.047075 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.047038    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k" event={"ID":"b25e1c06-7a83-41ed-ac61-096e603c6cdf","Type":"ContainerStarted","Data":"d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64"}
Sep 07 22:56:51.112218 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.112106    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" podUID="c8a2e723-b5c9-4d66-9249-be1844318063" containerName="main" containerID="cri-o://056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166" gracePeriod=45
Sep 07 22:56:51.399166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.399138    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:56:51.481460 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.481424    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:51.481631 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.481474    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:51.578395 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578365    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578441    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578484    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578558 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578517    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578702 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578567    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg8ng\" (UniqueName: \"kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578702 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578617    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache\") pod \"c8a2e723-b5c9-4d66-9249-be1844318063\" (UID: \"c8a2e723-b5c9-4d66-9249-be1844318063\") "
Sep 07 22:56:51.578823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578698    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home" (OuterVolumeSpecName: "home") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:51.578823 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578712    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:51.578936 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.578907    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:51.579042 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.579019    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:51.579086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.579046    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:51.579086 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.579055    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:51.580650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.580620    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm" (OuterVolumeSpecName: "dshm") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:51.580998 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.580977    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:56:51.581064 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.581019    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng" (OuterVolumeSpecName: "kube-api-access-jg8ng") pod "c8a2e723-b5c9-4d66-9249-be1844318063" (UID: "c8a2e723-b5c9-4d66-9249-be1844318063"). InnerVolumeSpecName "kube-api-access-jg8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:56:51.679607 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.679532    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a2e723-b5c9-4d66-9249-be1844318063-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:51.679607 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.679560    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8ng\" (UniqueName: \"kubernetes.io/projected/c8a2e723-b5c9-4d66-9249-be1844318063-kube-api-access-jg8ng\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:51.679607 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:51.679575    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a2e723-b5c9-4d66-9249-be1844318063-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:52.073469 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.073375    2550 generic.go:356] "Generic (PLEG): container finished" podID="c8a2e723-b5c9-4d66-9249-be1844318063" containerID="056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166" exitCode=0
Sep 07 22:56:52.073469 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.073454    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"
Sep 07 22:56:52.074004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.073464    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" event={"ID":"c8a2e723-b5c9-4d66-9249-be1844318063","Type":"ContainerDied","Data":"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"}
Sep 07 22:56:52.074004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.073526    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz" event={"ID":"c8a2e723-b5c9-4d66-9249-be1844318063","Type":"ContainerDied","Data":"7ffe76caab33c3671dc7d1549f363734c3b7e15039777e665a03ce0cd391ec8e"}
Sep 07 22:56:52.074004 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.073553    2550 scope.go:117] "RemoveContainer" containerID="056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"
Sep 07 22:56:52.087119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.087098    2550 scope.go:117] "RemoveContainer" containerID="056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"
Sep 07 22:56:52.087481 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:56:52.087451    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166\": container with ID starting with 056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166 not found: ID does not exist" containerID="056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"
Sep 07 22:56:52.087564 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.087483    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166"} err="failed to get container status \"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166\": rpc error: code = NotFound desc = could not find container \"056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166\": container with ID starting with 056b1ec6e80d863e581c615eed800ef8707bd69ed0cb89a2904b2fd4225e7166 not found: ID does not exist"
Sep 07 22:56:52.096120 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.096085    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:56:52.101426 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.101402    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/sub-test-model-7823cc-kserve-5656797867-fwlvz"]
Sep 07 22:56:52.186366 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.186334    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:52.186535 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.186383    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:52.199152 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.198976    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:52.229235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.229209    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:52.386754 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.386672    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca\") pod \"b25e1c06-7a83-41ed-ac61-096e603c6cdf\" (UID: \"b25e1c06-7a83-41ed-ac61-096e603c6cdf\") "
Sep 07 22:56:52.386993 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.386967    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b25e1c06-7a83-41ed-ac61-096e603c6cdf" (UID: "b25e1c06-7a83-41ed-ac61-096e603c6cdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:56:52.481529 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.481497    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 07 22:56:52.481687 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.481548    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 07 22:56:52.488190 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.488165    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e1c06-7a83-41ed-ac61-096e603c6cdf-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:56:52.796819 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:52.796779    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a2e723-b5c9-4d66-9249-be1844318063" path="/var/lib/kubelet/pods/c8a2e723-b5c9-4d66-9249-be1844318063/volumes"
Sep 07 22:56:53.080627 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.080543    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"
Sep 07 22:56:53.080627 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.080566    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k" event={"ID":"b25e1c06-7a83-41ed-ac61-096e603c6cdf","Type":"ContainerDied","Data":"d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64"}
Sep 07 22:56:53.080627 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.080611    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f8cf7cb93eae8d0a738685c348a20a9da9746e4be18d0a785a4e3795574d64"
Sep 07 22:56:53.093512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.093489    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:56:53.485360 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.485330    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:53.486339 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:53.486315    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:56:54.569442 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:56:54.569407    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:57:00.941940 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:00.941908    2550 scope.go:117] "RemoveContainer" containerID="b8c8126a9fafec4a101168f1c49c491e4b36a21385b7e99013535f6f97cb66ca"
Sep 07 22:57:04.938181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:04.938101    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:05.036110 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.036076    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:05.124748 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.124716    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"]
Sep 07 22:57:05.126954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.126920    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:05.127674 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.127653    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dd5b43-dxz7k"]
Sep 07 22:57:05.180074 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.180040    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:05.476965 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:05.476940    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:06.796778 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:06.796746    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25e1c06-7a83-41ed-ac61-096e603c6cdf" path="/var/lib/kubelet/pods/b25e1c06-7a83-41ed-ac61-096e603c6cdf/volumes"
Sep 07 22:57:09.363204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:09.363169    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:57:11.241177 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.241119    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:11.341830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.341786    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:11.491102 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.491069    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:57:11.491355 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.491285    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy" containerID="cri-o://683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093" gracePeriod=30
Sep 07 22:57:11.507677 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.507648    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:11.812029 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.811949    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"]
Sep 07 22:57:11.812425 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812407    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e1c06-7a83-41ed-ac61-096e603c6cdf" containerName="revoke-keys"
Sep 07 22:57:11.812425 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812425    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e1c06-7a83-41ed-ac61-096e603c6cdf" containerName="revoke-keys"
Sep 07 22:57:11.812569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812445    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a2e723-b5c9-4d66-9249-be1844318063" containerName="main"
Sep 07 22:57:11.812569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812451    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a2e723-b5c9-4d66-9249-be1844318063" containerName="main"
Sep 07 22:57:11.812569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812518    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e1c06-7a83-41ed-ac61-096e603c6cdf" containerName="revoke-keys"
Sep 07 22:57:11.812569 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.812528    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a2e723-b5c9-4d66-9249-be1844318063" containerName="main"
Sep 07 22:57:11.816765 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.816749    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:11.821005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.820984    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:57:11.821119 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.821077    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:57:11.826890 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.826865    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"]
Sep 07 22:57:11.848954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.848931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7776f-slb8q\" (UID: \"b6c393b7-2a37-4677-b857-c971b33633ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:11.950484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.950444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7776f-slb8q\" (UID: \"b6c393b7-2a37-4677-b857-c971b33633ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:11.951041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.951021    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c7776f-slb8q\" (UID: \"b6c393b7-2a37-4677-b857-c971b33633ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:11.987912 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:11.987852    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:12.127921 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:12.127825    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:12.397874 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:12.397843    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"]
Sep 07 22:57:12.399686 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:57:12.399657    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c393b7_2a37_4677_b857_c971b33633ab.slice/crio-44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4 WatchSource:0}: Error finding container 44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4: Status 404 returned error can't find the container with id 44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4
Sep 07 22:57:13.215151 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:13.215096    2550 generic.go:356] "Generic (PLEG): container finished" podID="b6c393b7-2a37-4677-b857-c971b33633ab" containerID="a06cea50097f1a6076b23bfa5e3800d03c246166f980593d7eac16ce47d45f43" exitCode=0
Sep 07 22:57:13.215362 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:13.215181    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q" event={"ID":"b6c393b7-2a37-4677-b857-c971b33633ab","Type":"ContainerDied","Data":"a06cea50097f1a6076b23bfa5e3800d03c246166f980593d7eac16ce47d45f43"}
Sep 07 22:57:13.215362 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:13.215264    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q" event={"ID":"b6c393b7-2a37-4677-b857-c971b33633ab","Type":"ContainerStarted","Data":"44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4"}
Sep 07 22:57:14.366690 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:14.366665    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:14.472597 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:14.472521    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca\") pod \"b6c393b7-2a37-4677-b857-c971b33633ab\" (UID: \"b6c393b7-2a37-4677-b857-c971b33633ab\") "
Sep 07 22:57:14.472830 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:14.472808    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6c393b7-2a37-4677-b857-c971b33633ab" (UID: "b6c393b7-2a37-4677-b857-c971b33633ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:14.574059 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:14.574026    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c393b7-2a37-4677-b857-c971b33633ab-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:15.227804 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:15.227771    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q" event={"ID":"b6c393b7-2a37-4677-b857-c971b33633ab","Type":"ContainerDied","Data":"44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4"}
Sep 07 22:57:15.227804 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:15.227800    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"
Sep 07 22:57:15.228002 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:15.227819    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b6802626b858c916959aa7c28b43b0044f6b7b67aaaeaac4086ae3a30fdca4"
Sep 07 22:57:16.742888 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.742863    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:57:16.793677 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793613    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793677 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793645    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793685    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793885 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793862    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793905    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793944    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.793997 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.793987    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8btr\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.794189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794040    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.794189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794099    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:16.794189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794117    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy\") pod \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\" (UID: \"1cc43396-bfe7-4cee-8ab8-edb244d026c6\") "
Sep 07 22:57:16.794189 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794139    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:16.794410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794337    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:16.794410 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794363    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:16.794592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794543    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:16.794592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794558    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.794592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794576    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.794592 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794589    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.794833 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.794602    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.796310 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.796289    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:57:16.796548 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.796525    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:16.796672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.796650    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:16.796792 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.796773    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr" (OuterVolumeSpecName: "kube-api-access-z8btr") pod "1cc43396-bfe7-4cee-8ab8-edb244d026c6" (UID: "1cc43396-bfe7-4cee-8ab8-edb244d026c6"). InnerVolumeSpecName "kube-api-access-z8btr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:16.895887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.895862    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.895887 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.895884    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.896036 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.895893    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc43396-bfe7-4cee-8ab8-edb244d026c6-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.896036 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.895902    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8btr\" (UniqueName: \"kubernetes.io/projected/1cc43396-bfe7-4cee-8ab8-edb244d026c6-kube-api-access-z8btr\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:16.896036 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:16.895915    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc43396-bfe7-4cee-8ab8-edb244d026c6-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:17.241547 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.241514    2550 generic.go:356] "Generic (PLEG): container finished" podID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerID="683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093" exitCode=0
Sep 07 22:57:17.241688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.241560    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"
Sep 07 22:57:17.241688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.241594    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerDied","Data":"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"}
Sep 07 22:57:17.241688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.241648    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn" event={"ID":"1cc43396-bfe7-4cee-8ab8-edb244d026c6","Type":"ContainerDied","Data":"2e3473faf0dbd9a503a4da3258fc785c0a641bb8317c97020599388a097d71f2"}
Sep 07 22:57:17.241688 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.241671    2550 scope.go:117] "RemoveContainer" containerID="683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"
Sep 07 22:57:17.250934 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.250917    2550 scope.go:117] "RemoveContainer" containerID="73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"
Sep 07 22:57:17.259872 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.259805    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:57:17.259936 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.259904    2550 scope.go:117] "RemoveContainer" containerID="683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"
Sep 07 22:57:17.260268 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:57:17.260233    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093\": container with ID starting with 683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093 not found: ID does not exist" containerID="683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"
Sep 07 22:57:17.260268 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.260270    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093"} err="failed to get container status \"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093\": rpc error: code = NotFound desc = could not find container \"683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093\": container with ID starting with 683d7de6b624cfe217547c8296708994dae19e5107e18939da880921bc7e7093 not found: ID does not exist"
Sep 07 22:57:17.260473 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.260294    2550 scope.go:117] "RemoveContainer" containerID="73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"
Sep 07 22:57:17.261003 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:57:17.260955    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471\": container with ID starting with 73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471 not found: ID does not exist" containerID="73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"
Sep 07 22:57:17.261003 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.260990    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471"} err="failed to get container status \"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471\": rpc error: code = NotFound desc = could not find container \"73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471\": container with ID starting with 73d0824a4f0b50ff6d891421f37cf956467d76ce638ceca6a78c2322f0325471 not found: ID does not exist"
Sep 07 22:57:17.263219 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.263200    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dd5b43-openshift-default-6b6c786bb4-9cbjn"]
Sep 07 22:57:17.742482 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.742453    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:17.810893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.810863    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:17.895651 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:17.895613    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:18.796121 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:18.796085    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" path="/var/lib/kubelet/pods/1cc43396-bfe7-4cee-8ab8-edb244d026c6/volumes"
Sep 07 22:57:24.273524 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:24.273442    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:57:27.449208 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:27.449170    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"]
Sep 07 22:57:27.451326 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:27.451302    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c7776f-slb8q"]
Sep 07 22:57:27.880429 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:27.880320    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:27.934096 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:27.934065    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:28.796671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:28.796635    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c393b7-2a37-4677-b857-c971b33633ab" path="/var/lib/kubelet/pods/b6c393b7-2a37-4677-b857-c971b33633ab/volumes"
Sep 07 22:57:33.571920 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:33.571868    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:33.658467 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:33.658431    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:33.762455 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:33.762422    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:57:33.762694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:33.762665    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" containerID="cri-o://0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d" gracePeriod=30
Sep 07 22:57:33.784273 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:33.784250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:34.114952 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.114917    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"]
Sep 07 22:57:34.115371 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115355    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:34.115440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115373    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:34.115440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115393    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c393b7-2a37-4677-b857-c971b33633ab" containerName="revoke-keys"
Sep 07 22:57:34.115440 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115398    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c393b7-2a37-4677-b857-c971b33633ab" containerName="revoke-keys"
Sep 07 22:57:34.115542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115483    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:34.115542 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.115494    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c393b7-2a37-4677-b857-c971b33633ab" containerName="revoke-keys"
Sep 07 22:57:34.118672 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.118652    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:34.121024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.121004    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:57:34.121154 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.121078    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:57:34.125953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.125929    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"]
Sep 07 22:57:34.247039 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.247011    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw\" (UID: \"cfa7f5ac-113f-4fb2-9160-59fa028c4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:34.303065 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.303041    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:34.347969 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.347943    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw\" (UID: \"cfa7f5ac-113f-4fb2-9160-59fa028c4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:34.348563 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.348545    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw\" (UID: \"cfa7f5ac-113f-4fb2-9160-59fa028c4375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:34.431112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.431036    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:34.699443 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:34.699417    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"]
Sep 07 22:57:34.702265 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:57:34.702228    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa7f5ac_113f_4fb2_9160_59fa028c4375.slice/crio-9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2 WatchSource:0}: Error finding container 9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2: Status 404 returned error can't find the container with id 9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2
Sep 07 22:57:35.344545 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:35.344506    2550 generic.go:356] "Generic (PLEG): container finished" podID="cfa7f5ac-113f-4fb2-9160-59fa028c4375" containerID="ec41a18ca6ead7f1bacf98d8a6de05eac191b33a4fe512b342e8df4a0593135b" exitCode=0
Sep 07 22:57:35.344670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:35.344571    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw" event={"ID":"cfa7f5ac-113f-4fb2-9160-59fa028c4375","Type":"ContainerDied","Data":"ec41a18ca6ead7f1bacf98d8a6de05eac191b33a4fe512b342e8df4a0593135b"}
Sep 07 22:57:35.344670 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:35.344625    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw" event={"ID":"cfa7f5ac-113f-4fb2-9160-59fa028c4375","Type":"ContainerStarted","Data":"9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2"}
Sep 07 22:57:36.486822 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:36.486798    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:36.568807 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:36.568780    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca\") pod \"cfa7f5ac-113f-4fb2-9160-59fa028c4375\" (UID: \"cfa7f5ac-113f-4fb2-9160-59fa028c4375\") "
Sep 07 22:57:36.569115 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:36.569094    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfa7f5ac-113f-4fb2-9160-59fa028c4375" (UID: "cfa7f5ac-113f-4fb2-9160-59fa028c4375"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:36.670456 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:36.670395    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7f5ac-113f-4fb2-9160-59fa028c4375-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:37.272608 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:37.272572    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:57:37.356991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:37.356964    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"
Sep 07 22:57:37.356991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:37.356973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw" event={"ID":"cfa7f5ac-113f-4fb2-9160-59fa028c4375","Type":"ContainerDied","Data":"9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2"}
Sep 07 22:57:37.357190 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:37.357019    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9979ad87c4fa7e05a4ab31d7d5ef843abc11aa56b86fb68c24b88e3875b299d2"
Sep 07 22:57:38.488765 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:38.488725    2550 patch_prober.go:28] interesting pod/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:57:38.489165 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:38.488783    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:57:39.019286 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.019264    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:57:39.192995 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.192967    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193014    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193037    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193066    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnm5b\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193095    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193113    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193175 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193167    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193237    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193271    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data\") pod \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\" (UID: \"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e\") "
Sep 07 22:57:39.193492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193280    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:39.193492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193320    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:39.193492 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193482    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:39.193755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193611    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:39.193755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193641    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:39.193755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193645    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.193755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193699    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.193755 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.193721    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.195361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.195338    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:39.195361 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.195354    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:57:39.195502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.195377    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:39.195502 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.195428    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b" (OuterVolumeSpecName: "kube-api-access-jnm5b") pod "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" (UID: "a8c36434-b9b8-4e53-9bc0-b62c5bd6175e"). InnerVolumeSpecName "kube-api-access-jnm5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:39.295041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295017    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.295041 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295039    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnm5b\" (UniqueName: \"kubernetes.io/projected/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-kube-api-access-jnm5b\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.295204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295049    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.295204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295058    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.295204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295067    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.295204 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.295074    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:39.373208 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.373179    2550 generic.go:356] "Generic (PLEG): container finished" podID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerID="0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d" exitCode=0
Sep 07 22:57:39.373312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.373222    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"
Sep 07 22:57:39.373312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.373235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerDied","Data":"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"}
Sep 07 22:57:39.373312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.373288    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k" event={"ID":"a8c36434-b9b8-4e53-9bc0-b62c5bd6175e","Type":"ContainerDied","Data":"cb799d2c59adb3e783925eb5d025b8da51b4e787a89e7c0787a1964586d849f1"}
Sep 07 22:57:39.373312 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.373311    2550 scope.go:117] "RemoveContainer" containerID="0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"
Sep 07 22:57:39.382966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.382940    2550 scope.go:117] "RemoveContainer" containerID="53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"
Sep 07 22:57:39.391512 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.391494    2550 scope.go:117] "RemoveContainer" containerID="0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"
Sep 07 22:57:39.391956 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:57:39.391825    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d\": container with ID starting with 0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d not found: ID does not exist" containerID="0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"
Sep 07 22:57:39.391956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.391862    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d"} err="failed to get container status \"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d\": rpc error: code = NotFound desc = could not find container \"0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d\": container with ID starting with 0003d44dfc4be181a6c4dedb791c360b7ffde6a9d074228a76ca4477388b674d not found: ID does not exist"
Sep 07 22:57:39.391956 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.391884    2550 scope.go:117] "RemoveContainer" containerID="53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"
Sep 07 22:57:39.392244 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:57:39.392219    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1\": container with ID starting with 53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1 not found: ID does not exist" containerID="53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"
Sep 07 22:57:39.392339 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.392251    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1"} err="failed to get container status \"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1\": rpc error: code = NotFound desc = could not find container \"53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1\": container with ID starting with 53b9be4a3d45e78699842a3c0d47cedc56f74dc423df9288cbe0dc07bfc0d0f1 not found: ID does not exist"
Sep 07 22:57:39.395287 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.395264    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:57:39.397616 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:39.397596    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c7776f-openshift-default-b5669f7b6-7lj2k"]
Sep 07 22:57:40.796427 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:40.796397    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" path="/var/lib/kubelet/pods/a8c36434-b9b8-4e53-9bc0-b62c5bd6175e/volumes"
Sep 07 22:57:44.556834 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.556802    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:44.615327 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.615289    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:44.716181 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.716146    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:44.732058 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.732026    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"]
Sep 07 22:57:44.736773 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.736747    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-82409b-krfsw"]
Sep 07 22:57:44.796643 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.796616    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa7f5ac-113f-4fb2-9160-59fa028c4375" path="/var/lib/kubelet/pods/cfa7f5ac-113f-4fb2-9160-59fa028c4375/volumes"
Sep 07 22:57:44.817151 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:44.817023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:45.071722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:45.071618    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:50.668616 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:50.668571    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:51.260235 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260202    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"]
Sep 07 22:57:51.260614 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260602    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260616    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260627    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa7f5ac-113f-4fb2-9160-59fa028c4375" containerName="revoke-keys"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260633    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa7f5ac-113f-4fb2-9160-59fa028c4375" containerName="revoke-keys"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260641    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260646    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260663 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260661    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260666    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260720    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260728    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc43396-bfe7-4cee-8ab8-edb244d026c6" containerName="istio-proxy"
Sep 07 22:57:51.260852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260735    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c36434-b9b8-4e53-9bc0-b62c5bd6175e" containerName="istio-proxy"
Sep 07 22:57:51.260852 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.260741    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa7f5ac-113f-4fb2-9160-59fa028c4375" containerName="revoke-keys"
Sep 07 22:57:51.265338 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.265322    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:51.268113 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.268083    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:57:51.268268 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.268093    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:57:51.270234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.270206    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"]
Sep 07 22:57:51.395304 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.395273    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4\" (UID: \"31c95df3-9578-4085-8a08-61d5582f376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:51.496218 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.496185    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4\" (UID: \"31c95df3-9578-4085-8a08-61d5582f376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:51.496773 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.496753    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4\" (UID: \"31c95df3-9578-4085-8a08-61d5582f376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:51.577954 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.577887    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:51.767203 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.767156    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:57:51.850570 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:51.850520    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"]
Sep 07 22:57:51.851457 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:57:51.851435    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c95df3_9578_4085_8a08_61d5582f376c.slice/crio-e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3 WatchSource:0}: Error finding container e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3: Status 404 returned error can't find the container with id e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3
Sep 07 22:57:52.459468 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:52.459428    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4" event={"ID":"31c95df3-9578-4085-8a08-61d5582f376c","Type":"ContainerStarted","Data":"b2d54f8f7d12c8325e12a7bc2601ffa1aeb0d8970c7de5d02a0b3017c7abadd7"}
Sep 07 22:57:52.459638 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:52.459492    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4" event={"ID":"31c95df3-9578-4085-8a08-61d5582f376c","Type":"ContainerStarted","Data":"e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3"}
Sep 07 22:57:52.474340 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:52.474290    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4" podStartSLOduration=1.474274849 podStartE2EDuration="1.474274849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:57:52.472885167 +0000 UTC m=+1552.315153435" watchObservedRunningTime="2026-09-07 22:57:52.474274849 +0000 UTC m=+1552.316543117"
Sep 07 22:57:53.469143 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:53.469083    2550 generic.go:356] "Generic (PLEG): container finished" podID="31c95df3-9578-4085-8a08-61d5582f376c" containerID="b2d54f8f7d12c8325e12a7bc2601ffa1aeb0d8970c7de5d02a0b3017c7abadd7" exitCode=0
Sep 07 22:57:53.469600 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:53.469152    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4" event={"ID":"31c95df3-9578-4085-8a08-61d5582f376c","Type":"ContainerDied","Data":"b2d54f8f7d12c8325e12a7bc2601ffa1aeb0d8970c7de5d02a0b3017c7abadd7"}
Sep 07 22:57:54.613191 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:54.613165    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:54.630947 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:54.630921    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca\") pod \"31c95df3-9578-4085-8a08-61d5582f376c\" (UID: \"31c95df3-9578-4085-8a08-61d5582f376c\") "
Sep 07 22:57:54.631352 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:54.631327    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31c95df3-9578-4085-8a08-61d5582f376c" (UID: "31c95df3-9578-4085-8a08-61d5582f376c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:54.732051 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:54.731985    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c95df3-9578-4085-8a08-61d5582f376c-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:57:55.482844 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:55.482804    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4" event={"ID":"31c95df3-9578-4085-8a08-61d5582f376c","Type":"ContainerDied","Data":"e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3"}
Sep 07 22:57:55.483085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:55.482863    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e432ba7905cfda7f86ecc173a7a5aa3ce9c76c42e3d62ae63080d8ba6a30ace3"
Sep 07 22:57:55.483085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:55.482830    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"
Sep 07 22:57:56.370035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:57:56.369996    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:58:00.289400 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:00.289364    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:58:00.307082 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:00.307046    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:00.970617 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:00.970588    2550 scope.go:117] "RemoveContainer" containerID="e461b6cbe94b87bb8a4687f6621fdc851773121d73a3d4cc2f1bcd822458bd55"
Sep 07 22:58:00.979020 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:00.978997    2550 scope.go:117] "RemoveContainer" containerID="ab36c9e22f8e3abe6d49996dd96dcaa749c15d9495a837c804f94958ea1185ce"
Sep 07 22:58:01.375720 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.375650    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:01.759806 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.759755    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:01.859356 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.859323    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:01.941202 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.941159    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"]
Sep 07 22:58:01.943561 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.943523    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:01.944260 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:01.944239    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5483b1-489f4"]
Sep 07 22:58:02.005886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.005855    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:02.057751 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.057675    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 22:58:02.058618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.058590    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c95df3-9578-4085-8a08-61d5582f376c" containerName="revoke-keys"
Sep 07 22:58:02.058778 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.058758    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c95df3-9578-4085-8a08-61d5582f376c" containerName="revoke-keys"
Sep 07 22:58:02.058907 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.058888    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c95df3-9578-4085-8a08-61d5582f376c" containerName="revoke-keys"
Sep 07 22:58:02.066565 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.066538    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.068990 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.068966    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1dd653-openshift-default-dockercfg-ftr9m\""
Sep 07 22:58:02.072613 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.072589    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 22:58:02.091386 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.091363    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:02.103557 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103533    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103577    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103652    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103690    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103736    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103786 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103760    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103899 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103838    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103899 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.103977 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.103910    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpmb\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.198498 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.198458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:02.204575 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204584    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204693    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204710 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204710    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204877 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204781    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204905    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpmb\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.204991 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204955    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205103 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.204997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205103 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.205061    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205239 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.205215    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205308 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.205286    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205404 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.205384    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.205584 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.205559    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.207038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.207015    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.207441 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.207424    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.212005 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.211984    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.213365 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.213343    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpmb\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb\") pod \"e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") " pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.299351 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.299315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:02.381457 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.381388    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:02.655876 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.655838    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 22:58:02.657348 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:58:02.657317    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a12c838_cb4f_4746_84aa_1fd1a83556b1.slice/crio-be1c843ff289ad96fe3c48d17ec9bfaf896e230ab323ec4b42a884c04bcaa4ba WatchSource:0}: Error finding container be1c843ff289ad96fe3c48d17ec9bfaf896e230ab323ec4b42a884c04bcaa4ba: Status 404 returned error can't find the container with id be1c843ff289ad96fe3c48d17ec9bfaf896e230ab323ec4b42a884c04bcaa4ba
Sep 07 22:58:02.659403 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.659370    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:02.659489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.659436    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:02.659489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.659464    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:02.798510 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:02.798474    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c95df3-9578-4085-8a08-61d5582f376c" path="/var/lib/kubelet/pods/31c95df3-9578-4085-8a08-61d5582f376c/volumes"
Sep 07 22:58:03.531505 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:03.531466    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerStarted","Data":"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"}
Sep 07 22:58:03.531910 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:03.531526    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerStarted","Data":"be1c843ff289ad96fe3c48d17ec9bfaf896e230ab323ec4b42a884c04bcaa4ba"}
Sep 07 22:58:03.554761 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:03.554714    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podStartSLOduration=1.5547000949999998 podStartE2EDuration="1.554700095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:03.55134664 +0000 UTC m=+1563.393614932" watchObservedRunningTime="2026-09-07 22:58:03.554700095 +0000 UTC m=+1563.396968361"
Sep 07 22:58:04.382491 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:04.382454    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:04.387198 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:04.387171    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:04.537212 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:04.537183    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:04.538085 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:04.538064    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:05.408013 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:05.407977    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:05.507540 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:05.507504    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:06.546976 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:06.546930    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:06.651243 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:06.651212    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:06.747088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:06.747062    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:07.023186 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:07.023152    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:07.122166 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:07.122110    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:08.351953 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:08.351920    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:08.452326 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:08.452267    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:08.589044 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:08.589011    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:58:08.589298 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:08.589270    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" containerID="cri-o://af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0" gracePeriod=30
Sep 07 22:58:08.612595 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:08.612524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:09.322686 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:09.322653    2550 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:58:09.322849 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:09.322707    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:58:11.370678 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:11.370642    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:58:13.885722 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:13.885695    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:58:14.012777 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012740    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.012777 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012785    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012801    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012816    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012833    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012860    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013035 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.012997    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013036    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013065    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5d9q\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013077    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013150    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket\") pod \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\" (UID: \"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b\") "
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013181    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:14.013307 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013276    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013403    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013538    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013560    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013573    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013587    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.013615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.013601    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.015112 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.015083    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:14.015296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.015271    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:14.015541 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.015515    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q" (OuterVolumeSpecName: "kube-api-access-x5d9q") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "kube-api-access-x5d9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:14.015615 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.015526    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" (UID: "8bf55f1c-1011-4e7f-bded-6050fe0ffc9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:58:14.114870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.114803    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.114870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.114825    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.114870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.114834    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.114870 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.114844    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5d9q\" (UniqueName: \"kubernetes.io/projected/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b-kube-api-access-x5d9q\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:14.596503 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.596469    2550 generic.go:356] "Generic (PLEG): container finished" podID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerID="af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0" exitCode=0
Sep 07 22:58:14.596671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.596537    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"
Sep 07 22:58:14.596671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.596551    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerDied","Data":"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"}
Sep 07 22:58:14.596671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.596628    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb" event={"ID":"8bf55f1c-1011-4e7f-bded-6050fe0ffc9b","Type":"ContainerDied","Data":"4bd0f12c482fe2f8db9344aedaea01cb19c792a8fcf3ddeabc6dd7c4b648edac"}
Sep 07 22:58:14.596671 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.596650    2550 scope.go:117] "RemoveContainer" containerID="af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"
Sep 07 22:58:14.605889 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.605866    2550 scope.go:117] "RemoveContainer" containerID="00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"
Sep 07 22:58:14.615793 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.615776    2550 scope.go:117] "RemoveContainer" containerID="af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"
Sep 07 22:58:14.616040 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:58:14.616022    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0\": container with ID starting with af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0 not found: ID does not exist" containerID="af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"
Sep 07 22:58:14.616088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.616048    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0"} err="failed to get container status \"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0\": rpc error: code = NotFound desc = could not find container \"af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0\": container with ID starting with af445896b2a6973a08c44e16a3454f811db8e70972fa5e57a9576dbf689363c0 not found: ID does not exist"
Sep 07 22:58:14.616088 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.616063    2550 scope.go:117] "RemoveContainer" containerID="00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"
Sep 07 22:58:14.616329 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:58:14.616313    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16\": container with ID starting with 00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16 not found: ID does not exist" containerID="00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"
Sep 07 22:58:14.616381 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.616332    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16"} err="failed to get container status \"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16\": rpc error: code = NotFound desc = could not find container \"00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16\": container with ID starting with 00427d10b1412494f4a886a70657511f30aabfd4bbde9d8c96fc0afb5dc8fc16 not found: ID does not exist"
Sep 07 22:58:14.620815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.620791    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:58:14.626828 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.626806    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5483b1-openshift-default-56794c4c6b-wkvsb"]
Sep 07 22:58:14.797649 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:14.797619    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" path="/var/lib/kubelet/pods/8bf55f1c-1011-4e7f-bded-6050fe0ffc9b/volumes"
Sep 07 22:58:15.309732 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.309686    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" podUID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" containerName="main" containerID="cri-o://c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3" gracePeriod=45
Sep 07 22:58:15.594598 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.594576    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:58:15.604839 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.604814    2550 generic.go:356] "Generic (PLEG): container finished" podID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" containerID="c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3" exitCode=0
Sep 07 22:58:15.604932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.604851    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"
Sep 07 22:58:15.604932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.604886    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" event={"ID":"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed","Type":"ContainerDied","Data":"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"}
Sep 07 22:58:15.604932 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.604929    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7" event={"ID":"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed","Type":"ContainerDied","Data":"a7248b3ab289d066a8eec2e5b965329a046b98708597b6dc38e98c764b3a8923"}
Sep 07 22:58:15.605038 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.604948    2550 scope.go:117] "RemoveContainer" containerID="c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"
Sep 07 22:58:15.614716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.614689    2550 scope.go:117] "RemoveContainer" containerID="c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"
Sep 07 22:58:15.615001 ip-10-0-143-210 kubenswrapper[2550]: E0907 22:58:15.614978    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3\": container with ID starting with c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3 not found: ID does not exist" containerID="c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"
Sep 07 22:58:15.615084 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.615012    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3"} err="failed to get container status \"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3\": rpc error: code = NotFound desc = could not find container \"c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3\": container with ID starting with c1cf11917db4d95f5fd5563c6256a3cde549b5eb39847f5582a70316b86ec3c3 not found: ID does not exist"
Sep 07 22:58:15.730198 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730156    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730384 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730321    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730384 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730354    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730389    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730410    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730484 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730439    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home\") pod \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\" (UID: \"2c405a8a-f0e4-4a57-bc04-4023ad5a43ed\") "
Sep 07 22:58:15.730650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730559    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:15.730650 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730607    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:15.730757 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730722    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home" (OuterVolumeSpecName: "home") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:15.730864 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730847    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.730915 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730893    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.730915 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.730907    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.732369 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.732342    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm" (OuterVolumeSpecName: "dshm") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:15.732479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.732411    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:58:15.732479 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.732431    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l" (OuterVolumeSpecName: "kube-api-access-n2w7l") pod "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" (UID: "2c405a8a-f0e4-4a57-bc04-4023ad5a43ed"). InnerVolumeSpecName "kube-api-access-n2w7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:15.832171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.832073    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.832171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.832098    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.832171 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.832107    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2w7l\" (UniqueName: \"kubernetes.io/projected/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed-kube-api-access-n2w7l\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 22:58:15.928430 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.928398    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:58:15.931815 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:15.931787    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/rate-test-model-1d0395-kserve-645bd9db7f-l6gq7"]
Sep 07 22:58:16.796622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:16.796578    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" path="/var/lib/kubelet/pods/2c405a8a-f0e4-4a57-bc04-4023ad5a43ed/volumes"
Sep 07 22:58:20.579297 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:20.579247    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:58:20.579694 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:20.579303    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:58:21.642920 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.642841    2550 generic.go:356] "Generic (PLEG): container finished" podID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerID="d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24" exitCode=0
Sep 07 22:58:21.643339 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.642914    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerDied","Data":"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"}
Sep 07 22:58:21.643389 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.643346    2550 scope.go:117] "RemoveContainer" containerID="d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"
Sep 07 22:58:21.644534 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.644501    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:21.644618 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.644597    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:21.644654 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:21.644628    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 22:58:22.654666 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:22.654621    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerStarted","Data":"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"}
Sep 07 22:58:23.381979 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:23.381942    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:23.382184 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:23.382110    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 07 22:58:23.382254 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:23.382176    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 07 22:58:24.381893 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:24.381858    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 07 22:58:24.382296 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:24.381914    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 07 22:58:25.382234 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:25.382195    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 07 22:58:25.382667 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:25.382264    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 07 22:58:26.382507 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:26.382463    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 07 22:58:26.382926 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:26.382528    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 07 22:58:27.382145 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:27.382089    2550 patch_prober.go:28] interesting pod/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 07 22:58:27.382411 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:27.382179    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 07 22:58:28.385489 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:28.385454    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:28.385949 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:28.385726    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:28.386493 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:28.386476    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 22:58:29.350914 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:29.350863    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:29.452409 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:29.452376    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:30.364292 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:30.364254    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:31.007225 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:31.007185    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:31.298740 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:31.298657    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:31.562687 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:31.562605    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:34.153839 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:34.153806    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:34.251485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:34.251451    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:35.325213 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:35.325179    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:35.425500 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:35.425467    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:35.525621 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:35.525577    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:35.772966 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:35.772935    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:35.864323 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:35.864290    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:38.875716 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:38.875675    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:58:51.977105 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:51.977069    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:52.074475 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:52.074440    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:52.544465 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:52.544433    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:56.232600 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:56.232568    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:56.287077 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:56.287039    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:58:59.133587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:58:59.133553    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:59:26.668300 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.668257    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 22:59:26.669336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669308    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 22:59:26.669336 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669338    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 22:59:26.669509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669389    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" containerName="main"
Sep 07 22:59:26.669509 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669399    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" containerName="main"
Sep 07 22:59:26.669647 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669632    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 22:59:26.669715 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.669666    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c405a8a-f0e4-4a57-bc04-4023ad5a43ed" containerName="main"
Sep 07 22:59:26.674089 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.674057    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.679622 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.679595    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1dd653\"/\"default-dockercfg-t2dkd\""
Sep 07 22:59:26.679855 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.679837    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1dd653\"/\"ipp-route-1dd653-kserve-self-signed-certs\""
Sep 07 22:59:26.680032 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.680017    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1dd653\"/\"kube-root-ca.crt\""
Sep 07 22:59:26.680661 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.680637    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1dd653\"/\"openshift-service-ca.crt\""
Sep 07 22:59:26.680977 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.680952    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 22:59:26.696037 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.696016    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:59:26.850203 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850168    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.850363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850226    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.850363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.850363 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850343    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.850487 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850405    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.850532 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.850490    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm597\" (UniqueName: \"kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951399 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951367    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm597\" (UniqueName: \"kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951473    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951500    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951587 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951518    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951805 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951643    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951851 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951821    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951886 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951846    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.951922 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.951901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.953814 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.953787    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.954099 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.954079    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.959024 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.959003    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm597\" (UniqueName: \"kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597\") pod \"ipp-route-1dd653-kserve-f87fbff95-bmthb\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") " pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:26.988485 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:26.988463    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:27.257098 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:27.257073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 22:59:27.258176 ip-10-0-143-210 kubenswrapper[2550]: W0907 22:59:27.258143    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa54e402_27d0_4b41_adff_362b2c93d2d7.slice/crio-536e100a56e3505546e7de7cf185241bfc7f39b85b47dd46e1f802e41fa53c94 WatchSource:0}: Error finding container 536e100a56e3505546e7de7cf185241bfc7f39b85b47dd46e1f802e41fa53c94: Status 404 returned error can't find the container with id 536e100a56e3505546e7de7cf185241bfc7f39b85b47dd46e1f802e41fa53c94
Sep 07 22:59:28.032497 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:28.032458    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" event={"ID":"fa54e402-27d0-4b41-adff-362b2c93d2d7","Type":"ContainerStarted","Data":"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"}
Sep 07 22:59:28.032857 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:28.032500    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" event={"ID":"fa54e402-27d0-4b41-adff-362b2c93d2d7","Type":"ContainerStarted","Data":"536e100a56e3505546e7de7cf185241bfc7f39b85b47dd46e1f802e41fa53c94"}
Sep 07 22:59:28.049736 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:28.049684    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" podStartSLOduration=2.049667252 podStartE2EDuration="2.049667252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:59:28.047182218 +0000 UTC m=+1647.889450486" watchObservedRunningTime="2026-09-07 22:59:28.049667252 +0000 UTC m=+1647.891935518"
Sep 07 22:59:32.473458 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:32.473412    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 22:59:36.149506 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:36.149469    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 22:59:36.989574 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:36.989545    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:36.989764 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:36.989590    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:37.002157 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:37.002134    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:37.093495 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:37.093470    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 22:59:57.469646 ip-10-0-143-210 kubenswrapper[2550]: I0907 22:59:57.469599    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:00:00.156209 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.156173    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"]
Sep 07 23:00:00.156661 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.156632    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 23:00:00.156661 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.156651    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 23:00:00.156737 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.156715    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf55f1c-1011-4e7f-bded-6050fe0ffc9b" containerName="istio-proxy"
Sep 07 23:00:00.160386 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.160365    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"
Sep 07 23:00:00.162105 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.162080    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"]
Sep 07 23:00:00.164264 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.164248    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:00:00.165836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.165817    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"]
Sep 07 23:00:00.165954 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.165936    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"
Sep 07 23:00:00.169374 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.169355    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"
Sep 07 23:00:00.171628 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.171612    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"
Sep 07 23:00:00.180581 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.180559    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"]
Sep 07 23:00:00.184399 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.184372    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"
Sep 07 23:00:00.184581 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.184568    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"]
Sep 07 23:00:00.187728 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.187701    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"]
Sep 07 23:00:00.231052 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.231018    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"
Sep 07 23:00:00.467169 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.467139    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"]
Sep 07 23:00:00.468726 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:00.468705    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe59aea1_fc32_47f4_a4e9_edc0dd88ff7e.slice/crio-39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e WatchSource:0}: Error finding container 39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e: Status 404 returned error can't find the container with id 39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e
Sep 07 23:00:00.474675 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.474654    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"]
Sep 07 23:00:00.476380 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:00.476360    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71eb2c8_81f5_4d1c_afcd_040d22d6bf67.slice/crio-2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d WatchSource:0}: Error finding container 2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d: Status 404 returned error can't find the container with id 2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d
Sep 07 23:00:00.529324 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:00.529301    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"]
Sep 07 23:00:00.530731 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:00.530703    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1763d85f_19e8_43d0_ad21_5d2bae8ce054.slice/crio-e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358 WatchSource:0}: Error finding container e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358: Status 404 returned error can't find the container with id e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358
Sep 07 23:00:01.038545 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.038477    2550 scope.go:117] "RemoveContainer" containerID="b91a30a6fa997b4abce4727655eff3257656d3c71bbd2281c75abac3d7c07f8b"
Sep 07 23:00:01.222030 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.221985    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv" event={"ID":"c71eb2c8-81f5-4d1c-afcd-040d22d6bf67","Type":"ContainerStarted","Data":"12665cfd3199fc4d15526b3c800959b5e67532fd71a3969d567bb89e397725f4"}
Sep 07 23:00:01.222518 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.222079    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv" event={"ID":"c71eb2c8-81f5-4d1c-afcd-040d22d6bf67","Type":"ContainerStarted","Data":"2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d"}
Sep 07 23:00:01.223658 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.223627    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9" event={"ID":"fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e","Type":"ContainerStarted","Data":"d16727a762570abab9faf0b66f3411ddfb35d68e4201c1960269c313a67bb88a"}
Sep 07 23:00:01.223762 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.223680    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9" event={"ID":"fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e","Type":"ContainerStarted","Data":"39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e"}
Sep 07 23:00:01.225181 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.225158    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd" event={"ID":"1763d85f-19e8-43d0-ad21-5d2bae8ce054","Type":"ContainerStarted","Data":"c65e4c114f4ed59a959598d9bdd223e511d80d0d4b125acc756223b383ab8ede"}
Sep 07 23:00:01.225284 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.225194    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd" event={"ID":"1763d85f-19e8-43d0-ad21-5d2bae8ce054","Type":"ContainerStarted","Data":"e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358"}
Sep 07 23:00:01.237921 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.237877    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv" podStartSLOduration=1.237863369 podStartE2EDuration="1.237863369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:01.235562022 +0000 UTC m=+1681.077830289" watchObservedRunningTime="2026-09-07 23:00:01.237863369 +0000 UTC m=+1681.080131636"
Sep 07 23:00:01.250109 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.250058    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9" podStartSLOduration=1.25004091 podStartE2EDuration="1.25004091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:01.248762634 +0000 UTC m=+1681.091030904" watchObservedRunningTime="2026-09-07 23:00:01.25004091 +0000 UTC m=+1681.092309177"
Sep 07 23:00:01.262964 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:01.262921    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd" podStartSLOduration=1.26290775 podStartE2EDuration="1.26290775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:01.262322157 +0000 UTC m=+1681.104590440" watchObservedRunningTime="2026-09-07 23:00:01.26290775 +0000 UTC m=+1681.105176016"
Sep 07 23:00:02.232884 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.232843    2550 generic.go:356] "Generic (PLEG): container finished" podID="fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e" containerID="d16727a762570abab9faf0b66f3411ddfb35d68e4201c1960269c313a67bb88a" exitCode=0
Sep 07 23:00:02.233350 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.232923    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9" event={"ID":"fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e","Type":"ContainerDied","Data":"d16727a762570abab9faf0b66f3411ddfb35d68e4201c1960269c313a67bb88a"}
Sep 07 23:00:02.234595 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.234567    2550 generic.go:356] "Generic (PLEG): container finished" podID="1763d85f-19e8-43d0-ad21-5d2bae8ce054" containerID="c65e4c114f4ed59a959598d9bdd223e511d80d0d4b125acc756223b383ab8ede" exitCode=0
Sep 07 23:00:02.234694 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.234643    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd" event={"ID":"1763d85f-19e8-43d0-ad21-5d2bae8ce054","Type":"ContainerDied","Data":"c65e4c114f4ed59a959598d9bdd223e511d80d0d4b125acc756223b383ab8ede"}
Sep 07 23:00:02.236381 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.236355    2550 generic.go:356] "Generic (PLEG): container finished" podID="c71eb2c8-81f5-4d1c-afcd-040d22d6bf67" containerID="12665cfd3199fc4d15526b3c800959b5e67532fd71a3969d567bb89e397725f4" exitCode=0
Sep 07 23:00:02.236491 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:02.236417    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv" event={"ID":"c71eb2c8-81f5-4d1c-afcd-040d22d6bf67","Type":"ContainerDied","Data":"12665cfd3199fc4d15526b3c800959b5e67532fd71a3969d567bb89e397725f4"}
Sep 07 23:00:03.409594 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:03.409568    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"
Sep 07 23:00:03.459872 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:03.459851    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"
Sep 07 23:00:03.462785 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:03.462762    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"
Sep 07 23:00:04.249068 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.249023    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"
Sep 07 23:00:04.249068 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.249036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9" event={"ID":"fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e","Type":"ContainerDied","Data":"39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e"}
Sep 07 23:00:04.249309 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.249087    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39eb5ae3118c7f7ce7e20de2ba6f52042a1f7d39456c6a34892adfa14ea0845e"
Sep 07 23:00:04.250513 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.250493    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"
Sep 07 23:00:04.250656 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.250575    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd" event={"ID":"1763d85f-19e8-43d0-ad21-5d2bae8ce054","Type":"ContainerDied","Data":"e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358"}
Sep 07 23:00:04.250656 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.250618    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e132dfee4b4734c8f3e75832607da77f3a866f8c4be0a99d53212fc34ac60358"
Sep 07 23:00:04.252013 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.251987    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv" event={"ID":"c71eb2c8-81f5-4d1c-afcd-040d22d6bf67","Type":"ContainerDied","Data":"2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d"}
Sep 07 23:00:04.252163 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.252021    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"
Sep 07 23:00:04.252163 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:04.252031    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc0dbfff5865e281a0652fb69c93dab214a38b134d34a0b755f9114ea68a48d"
Sep 07 23:00:05.076346 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.076309    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:05.334950 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.334858    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:05.335450 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335420    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71eb2c8-81f5-4d1c-afcd-040d22d6bf67" containerName="cleanup"
Sep 07 23:00:05.335450 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335446    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71eb2c8-81f5-4d1c-afcd-040d22d6bf67" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335481    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335491    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335503    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1763d85f-19e8-43d0-ad21-5d2bae8ce054" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335511    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1763d85f-19e8-43d0-ad21-5d2bae8ce054" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335603    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1763d85f-19e8-43d0-ad21-5d2bae8ce054" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335617    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71eb2c8-81f5-4d1c-afcd-040d22d6bf67" containerName="cleanup"
Sep 07 23:00:05.335651 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.335628    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e" containerName="cleanup"
Sep 07 23:00:05.339020 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.338998    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.341446 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.341426    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-277d3e-openshift-default-dockercfg-k5mkn\""
Sep 07 23:00:05.350845 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.350363    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:05.365195 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.365152    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:05.475083 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475048    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475294 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475092    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475294 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475113    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475294 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475173    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475294 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475465 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475312    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475465 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475343    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvfk\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475465 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475366    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.475465 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.475409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576722 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvfk\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576749    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576767    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576849    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.576893 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576873    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577241 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576899    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577241 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.576924    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577241 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.577168    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577413 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.577384    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577492 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.577466    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577553 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.577514    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.577669 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.577649    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.579215 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.579191    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.579713 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.579692    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.584302 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.584280    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.584302 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.584295    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvfk\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk\") pod \"e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") " pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.653671 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.653587    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:05.694983 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.694922    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:05.930552 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.930474    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:05.934488 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:05.934463    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2343af0f_e116_4e15_ba19_dd19ce622475.slice/crio-4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3 WatchSource:0}: Error finding container 4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3: Status 404 returned error can't find the container with id 4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3
Sep 07 23:00:05.937583 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.937549    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:05.937683 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.937610    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:05.937683 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:05.937641    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:06.265171 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:06.265121    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerStarted","Data":"6fab15edc277616c64877a9aa8b2e0c2f05f9503ec94a0c8f771750b14fdd295"}
Sep 07 23:00:06.265537 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:06.265183    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerStarted","Data":"4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3"}
Sep 07 23:00:06.284942 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:06.284888    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podStartSLOduration=1.2848707990000001 podStartE2EDuration="1.284870799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:06.282448037 +0000 UTC m=+1686.124716305" watchObservedRunningTime="2026-09-07 23:00:06.284870799 +0000 UTC m=+1686.127139066"
Sep 07 23:00:06.654676 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:06.654591    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:06.660680 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:06.660654    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:07.270950 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:07.270917    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:07.272014 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:07.271992    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:08.159703 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:08.159672    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:08.262345 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:08.262310    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:09.313565 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:09.313527    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:09.414000 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:09.413966    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:09.517170 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:09.517115    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:09.765750 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:09.765720    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:09.867002 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:09.866967    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:17.971576 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:17.971531    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:00:29.421886 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.421848    2550 generic.go:356] "Generic (PLEG): container finished" podID="2343af0f-e116-4e15-ba19-dd19ce622475" containerID="6fab15edc277616c64877a9aa8b2e0c2f05f9503ec94a0c8f771750b14fdd295" exitCode=0
Sep 07 23:00:29.422352 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.421919    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerDied","Data":"6fab15edc277616c64877a9aa8b2e0c2f05f9503ec94a0c8f771750b14fdd295"}
Sep 07 23:00:29.422352 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.422319    2550 scope.go:117] "RemoveContainer" containerID="6fab15edc277616c64877a9aa8b2e0c2f05f9503ec94a0c8f771750b14fdd295"
Sep 07 23:00:29.423387 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.423358    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:29.423493 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.423448    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:29.423493 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:29.423477    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 23:00:30.429578 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:30.429531    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerStarted","Data":"7a5c6f6ec69c7b81acbda5517847567a00bc72b6b55fb7fb4dd9ad94d0da46cb"}
Sep 07 23:00:30.654598 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:30.654560    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:30.654895 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:30.654793    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 07 23:00:30.655093 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:30.655067    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 07 23:00:31.654911 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:31.654865    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 07 23:00:31.655647 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:31.655612    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 07 23:00:32.654808 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:32.654762    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 07 23:00:32.655026 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:32.654848    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 07 23:00:33.136051 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.136013    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:33.654277 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.654236    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 07 23:00:33.654580 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.654550    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 07 23:00:33.792681 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.792650    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"]
Sep 07 23:00:33.795168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.795152    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:33.797408 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.797384    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:00:33.797534 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.797469    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:00:33.806777 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.806742    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"]
Sep 07 23:00:33.838102 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.838076    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs\" (UID: \"92b74304-8d76-40e6-8b60-a23041f644f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:33.938826 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.938795    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs\" (UID: \"92b74304-8d76-40e6-8b60-a23041f644f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:33.939463 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:33.939436    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs\" (UID: \"92b74304-8d76-40e6-8b60-a23041f644f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:34.109152 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:34.109087    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:34.403030 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:34.403002    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"]
Sep 07 23:00:34.404728 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:34.404696    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b74304_8d76_40e6_8b60_a23041f644f1.slice/crio-602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012 WatchSource:0}: Error finding container 602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012: Status 404 returned error can't find the container with id 602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012
Sep 07 23:00:34.464724 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:34.464651    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs" event={"ID":"92b74304-8d76-40e6-8b60-a23041f644f1","Type":"ContainerStarted","Data":"602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012"}
Sep 07 23:00:34.655539 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:34.655024    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 07 23:00:34.655539 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:34.655087    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 07 23:00:35.474415 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:35.474383    2550 generic.go:356] "Generic (PLEG): container finished" podID="92b74304-8d76-40e6-8b60-a23041f644f1" containerID="4bcf388b1d2a21fdcc57628c8184e9d88f62f8f20a9425ee2404b7b20984269a" exitCode=0
Sep 07 23:00:35.474810 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:35.474469    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs" event={"ID":"92b74304-8d76-40e6-8b60-a23041f644f1","Type":"ContainerDied","Data":"4bcf388b1d2a21fdcc57628c8184e9d88f62f8f20a9425ee2404b7b20984269a"}
Sep 07 23:00:35.960225 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:35.960192    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:35.960728 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:35.960702    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:35.962147 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:35.962106    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:36.610910 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:36.610886    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:36.668043 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:36.668012    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca\") pod \"92b74304-8d76-40e6-8b60-a23041f644f1\" (UID: \"92b74304-8d76-40e6-8b60-a23041f644f1\") "
Sep 07 23:00:36.668368 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:36.668347    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92b74304-8d76-40e6-8b60-a23041f644f1" (UID: "92b74304-8d76-40e6-8b60-a23041f644f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:36.769152 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:36.769051    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b74304-8d76-40e6-8b60-a23041f644f1-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:37.486847 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:37.486817    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"
Sep 07 23:00:37.487030 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:37.486847    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs" event={"ID":"92b74304-8d76-40e6-8b60-a23041f644f1","Type":"ContainerDied","Data":"602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012"}
Sep 07 23:00:37.487030 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:37.486901    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602e49a1a20ab1f243b1c85722144ac0749640e080dc64b8fbd6732be1045012"
Sep 07 23:00:39.883321 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:39.883282    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:39.984093 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:39.984055    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:40.069407 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.069369    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"]
Sep 07 23:00:40.072233 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.072207    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:40.072866 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.072844    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-277d3e-xwwcs"]
Sep 07 23:00:40.124622 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.124587    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:40.426270 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.426237    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:40.796822 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:40.796789    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b74304-8d76-40e6-8b60-a23041f644f1" path="/var/lib/kubelet/pods/92b74304-8d76-40e6-8b60-a23041f644f1/volumes"
Sep 07 23:00:46.093414 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:46.093376    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:46.149327 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:46.149293    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:46.339415 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:46.339379    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:46.339672 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:46.339644    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" containerID="cri-o://7a5c6f6ec69c7b81acbda5517847567a00bc72b6b55fb7fb4dd9ad94d0da46cb" gracePeriod=30
Sep 07 23:00:46.346400 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:46.346318    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:47.167113 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:47.167076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:00:49.814773 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.814736    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"]
Sep 07 23:00:49.815205 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.815191    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b74304-8d76-40e6-8b60-a23041f644f1" containerName="revoke-keys"
Sep 07 23:00:49.815274 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.815207    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b74304-8d76-40e6-8b60-a23041f644f1" containerName="revoke-keys"
Sep 07 23:00:49.815309 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.815290    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b74304-8d76-40e6-8b60-a23041f644f1" containerName="revoke-keys"
Sep 07 23:00:49.817927 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.817909    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:49.820221 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.820195    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:00:49.820221 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.820212    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:00:49.825323 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.825296    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"]
Sep 07 23:00:49.887840 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.887816    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn\" (UID: \"5d5540a0-cfba-4229-9a4d-eb9bc361dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:49.988705 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.988676    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn\" (UID: \"5d5540a0-cfba-4229-9a4d-eb9bc361dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:49.989323 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:49.989305    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn\" (UID: \"5d5540a0-cfba-4229-9a4d-eb9bc361dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:50.042307 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.042278    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:00:50.130409 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.130332    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:50.404334 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.404304    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"]
Sep 07 23:00:50.405505 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:00:50.405477    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5540a0_cfba_4229_9a4d_eb9bc361dff3.slice/crio-70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed WatchSource:0}: Error finding container 70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed: Status 404 returned error can't find the container with id 70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed
Sep 07 23:00:50.563579 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.563554    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn" event={"ID":"5d5540a0-cfba-4229-9a4d-eb9bc361dff3","Type":"ContainerStarted","Data":"70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed"}
Sep 07 23:00:50.962280 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.962244    2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 07 23:00:50.962641 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:50.962304    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 23:00:51.572300 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.572267    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d5540a0-cfba-4229-9a4d-eb9bc361dff3" containerID="cc5e9e03975a1702f4bfcad167e4ba78668ca5283c64405f49573e7eaec2ecd0" exitCode=0
Sep 07 23:00:51.572431 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.572360    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn" event={"ID":"5d5540a0-cfba-4229-9a4d-eb9bc361dff3","Type":"ContainerDied","Data":"cc5e9e03975a1702f4bfcad167e4ba78668ca5283c64405f49573e7eaec2ecd0"}
Sep 07 23:00:51.574194 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.574170    2550 generic.go:356] "Generic (PLEG): container finished" podID="2343af0f-e116-4e15-ba19-dd19ce622475" containerID="7a5c6f6ec69c7b81acbda5517847567a00bc72b6b55fb7fb4dd9ad94d0da46cb" exitCode=0
Sep 07 23:00:51.574292 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.574230    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerDied","Data":"7a5c6f6ec69c7b81acbda5517847567a00bc72b6b55fb7fb4dd9ad94d0da46cb"}
Sep 07 23:00:51.574292 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.574278    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw" event={"ID":"2343af0f-e116-4e15-ba19-dd19ce622475","Type":"ContainerDied","Data":"4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3"}
Sep 07 23:00:51.574371 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.574304    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4717316b057e7c9a1f92d4d4aaec14da7ff056f7d5cf09c7d164b8554355c7e3"
Sep 07 23:00:51.574371 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.574323    2550 scope.go:117] "RemoveContainer" containerID="6fab15edc277616c64877a9aa8b2e0c2f05f9503ec94a0c8f771750b14fdd295"
Sep 07 23:00:51.587506 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.587486    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:51.709617 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.709582    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710111 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710091    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710257 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710167    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710257 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710200    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710369 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxvfk\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710418 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710374    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710418 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710406    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710516 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710455    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710516 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token\") pod \"2343af0f-e116-4e15-ba19-dd19ce622475\" (UID: \"2343af0f-e116-4e15-ba19-dd19ce622475\") "
Sep 07 23:00:51.710829 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710799    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:00:51.710948 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.710904    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:00:51.711107 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.711085    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:00:51.711218 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.711182    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:51.711308 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.711289    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data" (OuterVolumeSpecName: "istio-data") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:00:51.712987 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.712946    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk" (OuterVolumeSpecName: "kube-api-access-cxvfk") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "kube-api-access-cxvfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:00:51.713253 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.713228    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:00:51.713565 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.713544    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 23:00:51.713634 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.713573    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token" (OuterVolumeSpecName: "istio-token") pod "2343af0f-e116-4e15-ba19-dd19ce622475" (UID: "2343af0f-e116-4e15-ba19-dd19ce622475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:00:51.811200 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811175    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811200 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811197    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811200 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811206    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811200 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811215    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2343af0f-e116-4e15-ba19-dd19ce622475-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811439 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811223    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvfk\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-kube-api-access-cxvfk\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811439 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811232    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2343af0f-e116-4e15-ba19-dd19ce622475-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811439 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811240    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811439 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811248    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2343af0f-e116-4e15-ba19-dd19ce622475-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:51.811439 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:51.811258    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2343af0f-e116-4e15-ba19-dd19ce622475-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:52.581626 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.581593    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"
Sep 07 23:00:52.613546 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.613508    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:52.617884 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.617858    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-277d3e-openshift-default-559cbdc476-kbqhw"]
Sep 07 23:00:52.720397 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.720373    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:52.797860 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.797830    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" path="/var/lib/kubelet/pods/2343af0f-e116-4e15-ba19-dd19ce622475/volumes"
Sep 07 23:00:52.819248 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.819227    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca\") pod \"5d5540a0-cfba-4229-9a4d-eb9bc361dff3\" (UID: \"5d5540a0-cfba-4229-9a4d-eb9bc361dff3\") "
Sep 07 23:00:52.819597 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.819575    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d5540a0-cfba-4229-9a4d-eb9bc361dff3" (UID: "5d5540a0-cfba-4229-9a4d-eb9bc361dff3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:52.920365 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:52.920292    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5540a0-cfba-4229-9a4d-eb9bc361dff3-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:00:53.589382 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:53.589344    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn" event={"ID":"5d5540a0-cfba-4229-9a4d-eb9bc361dff3","Type":"ContainerDied","Data":"70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed"}
Sep 07 23:00:53.589382 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:53.589372    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"
Sep 07 23:00:53.589790 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:00:53.589393    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70894f6a955adb4d01909197e60f1f1267632c22f55c679d623bfff5bedbe0ed"
Sep 07 23:01:05.402041 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.401980    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:05.460989 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.460953    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:05.566168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.566120    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:05.575728 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.575703    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"]
Sep 07 23:01:05.580397 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.580369    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1dd653-lhkqn"]
Sep 07 23:01:05.618277 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.618215    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:05.929809 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:05.929770    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:06.573670 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:06.573637    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:01:06.795784 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:06.795744    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5540a0-cfba-4229-9a4d-eb9bc361dff3" path="/var/lib/kubelet/pods/5d5540a0-cfba-4229-9a4d-eb9bc361dff3/volumes"
Sep 07 23:01:10.913386 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:10.913353    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 23:01:10.956679 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:10.956632    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:16.035654 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:16.035623    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:16.668519 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:16.668480    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:01:25.933776 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:25.933715    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" podUID="fa54e402-27d0-4b41-adff-362b2c93d2d7" containerName="main" containerID="cri-o://6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e" gracePeriod=45
Sep 07 23:01:26.187568 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.187511    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 23:01:26.293599 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293560    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.293783 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293616    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.293783 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293633    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.293783 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293670    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm597\" (UniqueName: \"kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.293783 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293703    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.293783 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293720    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir\") pod \"fa54e402-27d0-4b41-adff-362b2c93d2d7\" (UID: \"fa54e402-27d0-4b41-adff-362b2c93d2d7\") "
Sep 07 23:01:26.294067 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293886    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home" (OuterVolumeSpecName: "home") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:01:26.294067 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.293993    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:01:26.294195 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.294059    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:01:26.295768 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.295745    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm" (OuterVolumeSpecName: "dshm") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:01:26.295875 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.295804    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:01:26.295935 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.295903    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597" (OuterVolumeSpecName: "kube-api-access-tm597") pod "fa54e402-27d0-4b41-adff-362b2c93d2d7" (UID: "fa54e402-27d0-4b41-adff-362b2c93d2d7"). InnerVolumeSpecName "kube-api-access-tm597". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:01:26.394789 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394751    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-home\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.394789 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394785    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa54e402-27d0-4b41-adff-362b2c93d2d7-tls-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.394789 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394797    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm597\" (UniqueName: \"kubernetes.io/projected/fa54e402-27d0-4b41-adff-362b2c93d2d7-kube-api-access-tm597\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.395040 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394807    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-model-cache\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.395040 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394816    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-tmp-dir\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.395040 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.394823    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa54e402-27d0-4b41-adff-362b2c93d2d7-dshm\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:01:26.796862 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.796828    2550 generic.go:356] "Generic (PLEG): container finished" podID="fa54e402-27d0-4b41-adff-362b2c93d2d7" containerID="6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e" exitCode=0
Sep 07 23:01:26.797067 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.796922    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"
Sep 07 23:01:26.797260 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.797226    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" event={"ID":"fa54e402-27d0-4b41-adff-362b2c93d2d7","Type":"ContainerDied","Data":"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"}
Sep 07 23:01:26.797408 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.797268    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb" event={"ID":"fa54e402-27d0-4b41-adff-362b2c93d2d7","Type":"ContainerDied","Data":"536e100a56e3505546e7de7cf185241bfc7f39b85b47dd46e1f802e41fa53c94"}
Sep 07 23:01:26.797408 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.797289    2550 scope.go:117] "RemoveContainer" containerID="6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"
Sep 07 23:01:26.808004 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.807935    2550 scope.go:117] "RemoveContainer" containerID="6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"
Sep 07 23:01:26.808289 ip-10-0-143-210 kubenswrapper[2550]: E0907 23:01:26.808254    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e\": container with ID starting with 6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e not found: ID does not exist" containerID="6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"
Sep 07 23:01:26.808390 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.808295    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e"} err="failed to get container status \"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e\": rpc error: code = NotFound desc = could not find container \"6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e\": container with ID starting with 6eaa680815c826d76f49824aac7ff604a38603e7a4690396e051e1390b20080e not found: ID does not exist"
Sep 07 23:01:26.821749 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.821722    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 23:01:26.827548 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:26.827527    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1dd653/ipp-route-1dd653-kserve-f87fbff95-bmthb"]
Sep 07 23:01:28.796561 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:28.796520    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa54e402-27d0-4b41-adff-362b2c93d2d7" path="/var/lib/kubelet/pods/fa54e402-27d0-4b41-adff-362b2c93d2d7/volumes"
Sep 07 23:01:43.314358 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:43.314323    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:46.510059 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:46.510028    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:47.630615 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:47.630574    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:47.945339 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:47.945306    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:53.362316 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:53.362228    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:58.788891 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:58.788855    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:58.856896 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:58.856868    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:59.035981 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.035946    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 23:01:59.036274 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.036224    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy" containerID="cri-o://f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906" gracePeriod=30
Sep 07 23:01:59.065544 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.065442    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:59.397695 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.397611    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"]
Sep 07 23:01:59.398038 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398025    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:01:59.398082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398040    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:01:59.398082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398049    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa54e402-27d0-4b41-adff-362b2c93d2d7" containerName="main"
Sep 07 23:01:59.398082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398056    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa54e402-27d0-4b41-adff-362b2c93d2d7" containerName="main"
Sep 07 23:01:59.398082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398063    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5540a0-cfba-4229-9a4d-eb9bc361dff3" containerName="revoke-keys"
Sep 07 23:01:59.398082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398069    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5540a0-cfba-4229-9a4d-eb9bc361dff3" containerName="revoke-keys"
Sep 07 23:01:59.398266 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398146    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:01:59.398266 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398154    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa54e402-27d0-4b41-adff-362b2c93d2d7" containerName="main"
Sep 07 23:01:59.398266 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398162    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5540a0-cfba-4229-9a4d-eb9bc361dff3" containerName="revoke-keys"
Sep 07 23:01:59.398266 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.398169    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:01:59.401331 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.401314    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:01:59.404174 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.404146    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:01:59.404396 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.404370    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:01:59.407192 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.407170    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"]
Sep 07 23:01:59.476746 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.476707    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft\" (UID: \"f68f5b98-b667-4d01-ad70-ead1d5597164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:01:59.577533 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.577498    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft\" (UID: \"f68f5b98-b667-4d01-ad70-ead1d5597164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:01:59.578112 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.578092    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft\" (UID: \"f68f5b98-b667-4d01-ad70-ead1d5597164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:01:59.609356 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.609325    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:01:59.713061 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.713034    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:01:59.984102 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:01:59.984076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"]
Sep 07 23:02:01.003076 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:01.003044    2550 generic.go:356] "Generic (PLEG): container finished" podID="f68f5b98-b667-4d01-ad70-ead1d5597164" containerID="9bfa4911942f94be008630c452e32bb0ecce1132a2ece2dfd73652771a95c822" exitCode=0
Sep 07 23:02:01.003474 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:01.003146    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft" event={"ID":"f68f5b98-b667-4d01-ad70-ead1d5597164","Type":"ContainerDied","Data":"9bfa4911942f94be008630c452e32bb0ecce1132a2ece2dfd73652771a95c822"}
Sep 07 23:02:01.003474 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:01.003239    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft" event={"ID":"f68f5b98-b667-4d01-ad70-ead1d5597164","Type":"ContainerStarted","Data":"453d2f8fbfe02c70f1bc6ede081013853065f4ee89bdca2bba7ced59f97db9e3"}
Sep 07 23:02:02.144833 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:02.144808    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:02:02.302078 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:02.302000    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca\") pod \"f68f5b98-b667-4d01-ad70-ead1d5597164\" (UID: \"f68f5b98-b667-4d01-ad70-ead1d5597164\") "
Sep 07 23:02:02.302414 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:02.302392    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f68f5b98-b667-4d01-ad70-ead1d5597164" (UID: "f68f5b98-b667-4d01-ad70-ead1d5597164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:02.403006 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:02.402980    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f5b98-b667-4d01-ad70-ead1d5597164-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:03.017673 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:03.017642    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"
Sep 07 23:02:03.017840 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:03.017669    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft" event={"ID":"f68f5b98-b667-4d01-ad70-ead1d5597164","Type":"ContainerDied","Data":"453d2f8fbfe02c70f1bc6ede081013853065f4ee89bdca2bba7ced59f97db9e3"}
Sep 07 23:02:03.017840 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:03.017723    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453d2f8fbfe02c70f1bc6ede081013853065f4ee89bdca2bba7ced59f97db9e3"
Sep 07 23:02:04.298525 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.298474    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 23:02:04.415694 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415666    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415720    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415762    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415815    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415988 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415859    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415988 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415898    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415988 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415924    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrpmb\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415988 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415953    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.415988 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.415984    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs\") pod \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\" (UID: \"5a12c838-cb4f-4746-84aa-1fd1a83556b1\") "
Sep 07 23:02:04.416289 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416177    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:04.416289 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416222    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:04.416401 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416357    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:04.416401 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416379    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.416401 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416383    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:04.416401 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416398    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.416563 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.416484    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:04.417830 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.417805    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 23:02:04.418286 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.418257    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb" (OuterVolumeSpecName: "kube-api-access-wrpmb") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "kube-api-access-wrpmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:04.418286 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.418269    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:04.418420 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.418338    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a12c838-cb4f-4746-84aa-1fd1a83556b1" (UID: "5a12c838-cb4f-4746-84aa-1fd1a83556b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:04.516770 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516746    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516770 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516767    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516777    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516786    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrpmb\" (UniqueName: \"kubernetes.io/projected/5a12c838-cb4f-4746-84aa-1fd1a83556b1-kube-api-access-wrpmb\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516796    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516805    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a12c838-cb4f-4746-84aa-1fd1a83556b1-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:04.516902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:04.516813    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a12c838-cb4f-4746-84aa-1fd1a83556b1-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:05.031211 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.031175    2550 generic.go:356] "Generic (PLEG): container finished" podID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerID="f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906" exitCode=0
Sep 07 23:02:05.031342 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.031227    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerDied","Data":"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"}
Sep 07 23:02:05.031342 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.031276    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24" event={"ID":"5a12c838-cb4f-4746-84aa-1fd1a83556b1","Type":"ContainerDied","Data":"be1c843ff289ad96fe3c48d17ec9bfaf896e230ab323ec4b42a884c04bcaa4ba"}
Sep 07 23:02:05.031342 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.031301    2550 scope.go:117] "RemoveContainer" containerID="f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"
Sep 07 23:02:05.031342 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.031232    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"
Sep 07 23:02:05.040675 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.040658    2550 scope.go:117] "RemoveContainer" containerID="d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"
Sep 07 23:02:05.048824 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.048806    2550 scope.go:117] "RemoveContainer" containerID="f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"
Sep 07 23:02:05.049059 ip-10-0-143-210 kubenswrapper[2550]: E0907 23:02:05.049039    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906\": container with ID starting with f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906 not found: ID does not exist" containerID="f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"
Sep 07 23:02:05.049161 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.049072    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906"} err="failed to get container status \"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906\": rpc error: code = NotFound desc = could not find container \"f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906\": container with ID starting with f5b8030d11195b025ff2ed0c1a9a790a325ca4962e9f234f15122bea44e64906 not found: ID does not exist"
Sep 07 23:02:05.049161 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.049095    2550 scope.go:117] "RemoveContainer" containerID="d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"
Sep 07 23:02:05.049397 ip-10-0-143-210 kubenswrapper[2550]: E0907 23:02:05.049381    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24\": container with ID starting with d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24 not found: ID does not exist" containerID="d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"
Sep 07 23:02:05.049442 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.049403    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24"} err="failed to get container status \"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24\": rpc error: code = NotFound desc = could not find container \"d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24\": container with ID starting with d014f3adb68a0b772e3e3ef3a96df04deb232fb3e75523b15a20b494d1ab7c24 not found: ID does not exist"
Sep 07 23:02:05.060025 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.059975    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 23:02:05.070466 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:05.070445    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dd653-openshift-default-7879c56c76-sqt24"]
Sep 07 23:02:06.796936 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:06.796900    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" path="/var/lib/kubelet/pods/5a12c838-cb4f-4746-84aa-1fd1a83556b1/volumes"
Sep 07 23:02:09.849737 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:09.849702    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:09.851700 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:09.851679    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"]
Sep 07 23:02:09.855353 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:09.855327    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-046f93-29813700-2jlhd"]
Sep 07 23:02:09.912800 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:09.912752    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:09.973016 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:09.972987    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:10.022264 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.022233    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"]
Sep 07 23:02:10.024300 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.024277    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-046f93-qrpft"]
Sep 07 23:02:10.073334 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.073311    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:10.387830 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.387797    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:10.795972 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.795937    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1763d85f-19e8-43d0-ad21-5d2bae8ce054" path="/var/lib/kubelet/pods/1763d85f-19e8-43d0-ad21-5d2bae8ce054/volumes"
Sep 07 23:02:10.796273 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:10.796259    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f68f5b98-b667-4d01-ad70-ead1d5597164" path="/var/lib/kubelet/pods/f68f5b98-b667-4d01-ad70-ead1d5597164/volumes"
Sep 07 23:02:12.377998 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:12.377959    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:02:15.884424 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:15.884390    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:15.934773 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:15.934745    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:16.136525 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.136450    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:16.488175 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488139    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"]
Sep 07 23:02:16.488540 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488527    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488542    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488554    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488559    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2343af0f-e116-4e15-ba19-dd19ce622475" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488567    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488572    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488584    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68f5b98-b667-4d01-ad70-ead1d5597164" containerName="revoke-keys"
Sep 07 23:02:16.488588 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488590    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68f5b98-b667-4d01-ad70-ead1d5597164" containerName="revoke-keys"
Sep 07 23:02:16.488809 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488654    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68f5b98-b667-4d01-ad70-ead1d5597164" containerName="revoke-keys"
Sep 07 23:02:16.488809 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488666    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.488809 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.488672    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a12c838-cb4f-4746-84aa-1fd1a83556b1" containerName="istio-proxy"
Sep 07 23:02:16.493450 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.493432    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:16.496437 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.496412    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:02:16.496558 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.496412    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:02:16.498583 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.498559    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"]
Sep 07 23:02:16.599620 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.599591    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:16.619395 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.619365    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb\" (UID: \"f4c7cd50-8620-43ba-868a-ac3d0148d7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:16.720694 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.720655    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb\" (UID: \"f4c7cd50-8620-43ba-868a-ac3d0148d7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:16.721368 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.721347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb\" (UID: \"f4c7cd50-8620-43ba-868a-ac3d0148d7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:16.807234 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:16.807170    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:17.076768 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:17.076745    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"]
Sep 07 23:02:17.078650 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:02:17.078621    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c7cd50_8620_43ba_868a_ac3d0148d7ba.slice/crio-9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8 WatchSource:0}: Error finding container 9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8: Status 404 returned error can't find the container with id 9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8
Sep 07 23:02:17.106725 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:17.106692    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb" event={"ID":"f4c7cd50-8620-43ba-868a-ac3d0148d7ba","Type":"ContainerStarted","Data":"9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8"}
Sep 07 23:02:18.114766 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:18.114676    2550 generic.go:356] "Generic (PLEG): container finished" podID="f4c7cd50-8620-43ba-868a-ac3d0148d7ba" containerID="0171cb6ae7dd9e130fc994d2a5d525b961040c527211593568a091342e70fac6" exitCode=0
Sep 07 23:02:18.115104 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:18.114761    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb" event={"ID":"f4c7cd50-8620-43ba-868a-ac3d0148d7ba","Type":"ContainerDied","Data":"0171cb6ae7dd9e130fc994d2a5d525b961040c527211593568a091342e70fac6"}
Sep 07 23:02:19.254896 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:19.254874    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:19.441172 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:19.441123    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca\") pod \"f4c7cd50-8620-43ba-868a-ac3d0148d7ba\" (UID: \"f4c7cd50-8620-43ba-868a-ac3d0148d7ba\") "
Sep 07 23:02:19.441487 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:19.441462    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4c7cd50-8620-43ba-868a-ac3d0148d7ba" (UID: "f4c7cd50-8620-43ba-868a-ac3d0148d7ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:19.542470 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:19.542441    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c7cd50-8620-43ba-868a-ac3d0148d7ba-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:20.128939 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:20.128907    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"
Sep 07 23:02:20.129104 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:20.128942    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb" event={"ID":"f4c7cd50-8620-43ba-868a-ac3d0148d7ba","Type":"ContainerDied","Data":"9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8"}
Sep 07 23:02:20.129104 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:20.128993    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c345cd8c0ad3870615e55003a1c31549322787033ce4411d62b82e01b7e29f8"
Sep 07 23:02:26.929261 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:26.929221    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"]
Sep 07 23:02:26.931370 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:26.931344    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d0395-29813700-wt6kv"]
Sep 07 23:02:26.937433 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:26.937404    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:26.995468 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:26.995439    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:27.095320 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:27.095291    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:27.130309 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:27.130277    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"]
Sep 07 23:02:27.132276 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:27.132256    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d0395-xchtb"]
Sep 07 23:02:27.154204 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:27.154178    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:27.511961 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:27.511925    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:28.796399 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:28.796365    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71eb2c8-81f5-4d1c-afcd-040d22d6bf67" path="/var/lib/kubelet/pods/c71eb2c8-81f5-4d1c-afcd-040d22d6bf67/volumes"
Sep 07 23:02:28.796772 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:28.796655    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c7cd50-8620-43ba-868a-ac3d0148d7ba" path="/var/lib/kubelet/pods/f4c7cd50-8620-43ba-868a-ac3d0148d7ba/volumes"
Sep 07 23:02:30.879494 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:30.879463    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:02:33.313489 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.313447    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:33.406990 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.406948    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:33.537636 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.537597    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 23:02:33.537943 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.537911    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy" containerID="cri-o://9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767" gracePeriod=30
Sep 07 23:02:33.561113 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.561086    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:33.890840 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.890806    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"]
Sep 07 23:02:33.891245 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.891231    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c7cd50-8620-43ba-868a-ac3d0148d7ba" containerName="revoke-keys"
Sep 07 23:02:33.891291 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.891247    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c7cd50-8620-43ba-868a-ac3d0148d7ba" containerName="revoke-keys"
Sep 07 23:02:33.891338 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.891329    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c7cd50-8620-43ba-868a-ac3d0148d7ba" containerName="revoke-keys"
Sep 07 23:02:33.894410 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.894393    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:33.896948 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.896925    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:02:33.897056 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.896956    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:02:33.901211 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.901107    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"]
Sep 07 23:02:33.965273 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:33.965241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr\" (UID: \"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:34.004942 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:34.004915    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:34.065854 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:34.065828    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr\" (UID: \"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:34.066493 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:34.066470    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr\" (UID: \"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:34.207550 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:34.207520    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:34.478182 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:34.478156    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"]
Sep 07 23:02:34.479475 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:02:34.479446    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e9d457_9dc7_49f3_ad17_3f65e9eb7fa4.slice/crio-14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a WatchSource:0}: Error finding container 14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a: Status 404 returned error can't find the container with id 14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a
Sep 07 23:02:35.221010 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:35.220974    2550 generic.go:356] "Generic (PLEG): container finished" podID="a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" containerID="89330f5cee28da6764819a1efe8d61996cc18649473a86a6b0a01b563dec4d09" exitCode=0
Sep 07 23:02:35.221256 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:35.221031    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr" event={"ID":"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4","Type":"ContainerDied","Data":"89330f5cee28da6764819a1efe8d61996cc18649473a86a6b0a01b563dec4d09"}
Sep 07 23:02:35.221256 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:35.221085    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr" event={"ID":"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4","Type":"ContainerStarted","Data":"14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a"}
Sep 07 23:02:36.364464 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:36.364439    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:36.489486 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:36.489403    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca\") pod \"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4\" (UID: \"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4\") "
Sep 07 23:02:36.489760 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:36.489737    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" (UID: "a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:36.589932 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:36.589905    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:37.234119 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:37.234090    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"
Sep 07 23:02:37.234306 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:37.234095    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr" event={"ID":"a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4","Type":"ContainerDied","Data":"14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a"}
Sep 07 23:02:37.234306 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:37.234178    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14bc3f865d5276c43aaa1636af413911431a9fe37eda6893d687fe4c4f889d7a"
Sep 07 23:02:38.781350 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.781322    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 23:02:38.907442 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907369    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907442 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907409    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907442 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907433    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907450    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907472    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907513    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sjjq\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907537    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907592    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907633    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert\") pod \"9649c969-f473-4263-8fb1-acbcd941fe07\" (UID: \"9649c969-f473-4263-8fb1-acbcd941fe07\") "
Sep 07 23:02:38.907709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907676    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907860    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data" (OuterVolumeSpecName: "istio-data") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.907892    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.908035    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.908055    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.908071    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:38.908084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.908053    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:38.908427 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.908309    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:38.909665 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.909643    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token" (OuterVolumeSpecName: "istio-token") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:38.909825 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.909805    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq" (OuterVolumeSpecName: "kube-api-access-6sjjq") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "kube-api-access-6sjjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:38.910072 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.910043    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 23:02:38.910072 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:38.910059    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9649c969-f473-4263-8fb1-acbcd941fe07" (UID: "9649c969-f473-4263-8fb1-acbcd941fe07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:39.008778 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008748    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.008778 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008775    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9649c969-f473-4263-8fb1-acbcd941fe07-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.008778 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008784    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.008974 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008792    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9649c969-f473-4263-8fb1-acbcd941fe07-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.008974 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008800    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9649c969-f473-4263-8fb1-acbcd941fe07-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.008974 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.008808    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjjq\" (UniqueName: \"kubernetes.io/projected/9649c969-f473-4263-8fb1-acbcd941fe07-kube-api-access-6sjjq\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:39.248773 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.248741    2550 generic.go:356] "Generic (PLEG): container finished" podID="9649c969-f473-4263-8fb1-acbcd941fe07" containerID="9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767" exitCode=0
Sep 07 23:02:39.248947 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.248787    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"
Sep 07 23:02:39.248947 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.248818    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerDied","Data":"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"}
Sep 07 23:02:39.248947 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.248861    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f" event={"ID":"9649c969-f473-4263-8fb1-acbcd941fe07","Type":"ContainerDied","Data":"0fdefcae3b1f6acd25dc1afc523422d3a2ea4cc2d548099a7a5fa97fda6a7816"}
Sep 07 23:02:39.248947 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.248883    2550 scope.go:117] "RemoveContainer" containerID="9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"
Sep 07 23:02:39.259071 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.259055    2550 scope.go:117] "RemoveContainer" containerID="509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.268350    2550 scope.go:117] "RemoveContainer" containerID="9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: E0907 23:02:39.268665    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767\": container with ID starting with 9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767 not found: ID does not exist" containerID="9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.268716    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767"} err="failed to get container status \"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767\": rpc error: code = NotFound desc = could not find container \"9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767\": container with ID starting with 9ae44f18ffc130a9be558f74b599eecd3e6bd1256bf759506b5e5055c9f3d767 not found: ID does not exist"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.268740    2550 scope.go:117] "RemoveContainer" containerID="509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: E0907 23:02:39.269114    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d\": container with ID starting with 509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d not found: ID does not exist" containerID="509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"
Sep 07 23:02:39.269168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.269155    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d"} err="failed to get container status \"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d\": rpc error: code = NotFound desc = could not find container \"509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d\": container with ID starting with 509bc1a89af80e8575585e108d02941122cbec04066362ab0213792baf6d751d not found: ID does not exist"
Sep 07 23:02:39.271313 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.271288    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 23:02:39.273412 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:39.273391    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d0395-openshift-default-ffc595dcb-htz8f"]
Sep 07 23:02:40.796661 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:40.796619    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" path="/var/lib/kubelet/pods/9649c969-f473-4263-8fb1-acbcd941fe07/volumes"
Sep 07 23:02:44.317723 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.317678    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:44.318614 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.318587    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"]
Sep 07 23:02:44.322198 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.322171    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7823cc-29813700-lptq9"]
Sep 07 23:02:44.380925 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.380879    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:44.439038 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.438994    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:44.497027 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.496997    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"]
Sep 07 23:02:44.499951 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.499924    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7823cc-cwrsr"]
Sep 07 23:02:44.501424 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.501388    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:44.554559 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.554503    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:44.795834 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.795800    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" path="/var/lib/kubelet/pods/a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4/volumes"
Sep 07 23:02:44.796098 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.796085    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e" path="/var/lib/kubelet/pods/fe59aea1-fc32-47f4-a4e9-edc0dd88ff7e/volumes"
Sep 07 23:02:44.864639 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:44.864608    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:46.867533 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:46.867498    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:02:50.963738 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:50.963691    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:51.015497 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:51.015449    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:51.188011 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:51.187974    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 23:02:51.188292 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:51.188264    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy" containerID="cri-o://30482e9ffedd3bc8708d36411254dd77ca4fc1341715a7057c2663e8da3fc881" gracePeriod=30
Sep 07 23:02:51.211114 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:51.211078    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:02:56.355115 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.355078    2550 generic.go:356] "Generic (PLEG): container finished" podID="385b048f-c008-42a9-811f-b9de2c5d8735" containerID="30482e9ffedd3bc8708d36411254dd77ca4fc1341715a7057c2663e8da3fc881" exitCode=0
Sep 07 23:02:56.355523 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.355155    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerDied","Data":"30482e9ffedd3bc8708d36411254dd77ca4fc1341715a7057c2663e8da3fc881"}
Sep 07 23:02:56.355523 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.355232    2550 scope.go:117] "RemoveContainer" containerID="19f1aead48959b59e6580364e18d5fbdcc23e591f5f77bd67faae04956c67862"
Sep 07 23:02:56.444376 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.444355    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 23:02:56.554103 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554040    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554103 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554092    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554276 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554120    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554276 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554202    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554276 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554262    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554428 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554296    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfjnh\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554428 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554333    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:56.554428 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554367    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554586 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554438    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:56.554586 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554465    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:56.554586 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554586 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554550    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy\") pod \"385b048f-c008-42a9-811f-b9de2c5d8735\" (UID: \"385b048f-c008-42a9-811f-b9de2c5d8735\") "
Sep 07 23:02:56.554824 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554534    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data" (OuterVolumeSpecName: "istio-data") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:56.554896 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554876    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.554962 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554904    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.554962 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554917    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.554962 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.554932    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.555168 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.555113    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:56.556556 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.556529    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 23:02:56.556556 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.556546    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh" (OuterVolumeSpecName: "kube-api-access-tfjnh") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "kube-api-access-tfjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:56.556556 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.556551    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:56.556735 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.556589    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token" (OuterVolumeSpecName: "istio-token") pod "385b048f-c008-42a9-811f-b9de2c5d8735" (UID: "385b048f-c008-42a9-811f-b9de2c5d8735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:56.655796 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.655772    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.655796 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.655797    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfjnh\" (UniqueName: \"kubernetes.io/projected/385b048f-c008-42a9-811f-b9de2c5d8735-kube-api-access-tfjnh\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.655945 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.655812    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385b048f-c008-42a9-811f-b9de2c5d8735-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.655945 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.655825    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385b048f-c008-42a9-811f-b9de2c5d8735-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:56.655945 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:56.655836    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385b048f-c008-42a9-811f-b9de2c5d8735-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\""
Sep 07 23:02:57.362644 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.362609    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"
Sep 07 23:02:57.363042 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.362611    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn" event={"ID":"385b048f-c008-42a9-811f-b9de2c5d8735","Type":"ContainerDied","Data":"9cc9d657bea515cfb5db30ebfc4d04e4128ffd135cfc59cb89e158ae28c43b07"}
Sep 07 23:02:57.363042 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.362769    2550 scope.go:117] "RemoveContainer" containerID="30482e9ffedd3bc8708d36411254dd77ca4fc1341715a7057c2663e8da3fc881"
Sep 07 23:02:57.380418 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.380389    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 23:02:57.382269 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.382247    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7823cc-openshift-default-577b59f958-h8xpn"]
Sep 07 23:02:57.562974 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:57.562946    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:02:58.796885 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:02:58.796845    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" path="/var/lib/kubelet/pods/385b048f-c008-42a9-811f-b9de2c5d8735/volumes"
Sep 07 23:03:01.089180 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:03:01.089150    2550 scope.go:117] "RemoveContainer" containerID="f9e922cc92b1ff7add1019150d96cfeb93b48bfe8c09b01780369c5c6ac7e34a"
Sep 07 23:03:01.098116 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:03:01.098097    2550 scope.go:117] "RemoveContainer" containerID="cede1f773fa867dc481f2dd3cb652b02f217cedac1ed30aa58db4079cee20d03"
Sep 07 23:03:12.968084 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:03:12.968051    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:03:14.199297 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:03:14.199265    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:04:01.141839 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:01.141807    2550 scope.go:117] "RemoveContainer" containerID="ec41a18ca6ead7f1bacf98d8a6de05eac191b33a4fe512b342e8df4a0593135b"
Sep 07 23:04:01.150458 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:01.150435    2550 scope.go:117] "RemoveContainer" containerID="b2d54f8f7d12c8325e12a7bc2601ffa1aeb0d8970c7de5d02a0b3017c7abadd7"
Sep 07 23:04:01.158573 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:01.158553    2550 scope.go:117] "RemoveContainer" containerID="a06cea50097f1a6076b23bfa5e3800d03c246166f980593d7eac16ce47d45f43"
Sep 07 23:04:14.685032 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:14.684993    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:04:15.567536 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:15.567497    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:04:15.913836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:15.913759    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:04:16.594082 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:16.594045    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:04:18.150832 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:18.150792    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:04:40.069083 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:40.069047    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:04:47.969654 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:47.969618    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:04:57.262180 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:04:57.262086    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:05:04.867514 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:05:04.867466    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:05:15.467948 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:05:15.467910    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:05:21.167304 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:05:21.167264    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:06:01.194736 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:06:01.194698    2550 scope.go:117] "RemoveContainer" containerID="c65e4c114f4ed59a959598d9bdd223e511d80d0d4b125acc756223b383ab8ede"
Sep 07 23:06:01.203395 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:06:01.203376    2550 scope.go:117] "RemoveContainer" containerID="d16727a762570abab9faf0b66f3411ddfb35d68e4201c1960269c313a67bb88a"
Sep 07 23:06:01.217786 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:06:01.217761    2550 scope.go:117] "RemoveContainer" containerID="12665cfd3199fc4d15526b3c800959b5e67532fd71a3969d567bb89e397725f4"
Sep 07 23:06:46.961157 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:06:46.961104    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:06:55.066832 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:06:55.066791    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:07:01.249014 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:01.248977    2550 scope.go:117] "RemoveContainer" containerID="cc5e9e03975a1702f4bfcad167e4ba78668ca5283c64405f49573e7eaec2ecd0"
Sep 07 23:07:01.257796 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:01.257777    2550 scope.go:117] "RemoveContainer" containerID="4bcf388b1d2a21fdcc57628c8184e9d88f62f8f20a9425ee2404b7b20984269a"
Sep 07 23:07:01.265412 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:01.265389    2550 scope.go:117] "RemoveContainer" containerID="7a5c6f6ec69c7b81acbda5517847567a00bc72b6b55fb7fb4dd9ad94d0da46cb"
Sep 07 23:07:04.322267 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:04.322219    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:06.864766 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:06.864729    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:07:13.676335 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:13.676296    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:19.419780 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:19.419742    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:19.466359 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:19.466320    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:20.495253 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:20.495221    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:23.163007 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:23.162971    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:07:46.314001 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:46.313946    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:48.102288 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:48.102246    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:07:48.461013 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:48.460977    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:07:56.163420 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:07:56.163337    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:01.293954 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:01.293923    2550 scope.go:117] "RemoveContainer" containerID="9bfa4911942f94be008630c452e32bb0ecce1132a2ece2dfd73652771a95c822"
Sep 07 23:08:04.325372 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:04.325338    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:11.967188 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:11.967152    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:20.260934 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:20.260900    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:26.060836 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:26.060801    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:33.875800 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:33.875758    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lv56r"]
Sep 07 23:08:40.891066 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:40.891034    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s2564_429d30f9-8d79-490b-8045-139ee29171a6/manager/0.log"
Sep 07 23:08:41.231282 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:41.231253    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tw2sg_b22bcf05-840c-49b5-bcd5-e896a4419835/server/0.log"
Sep 07 23:08:41.339487 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:41.339456    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtn7q_e1bd5a87-03ac-47d2-b817-a351191470d1/manager/0.log"
Sep 07 23:08:41.446431 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:41.446402    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-btp2q_7da6ed5b-70f7-4893-9061-abf8eb5cde9b/manager/0.log"
Sep 07 23:08:41.451999 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:41.451980    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-btp2q_7da6ed5b-70f7-4893-9061-abf8eb5cde9b/copy-manifests/0.log"
Sep 07 23:08:43.263642 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:43.263603    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-96z4x_5a0c0620-609d-4df0-872a-b383bec31142/manager/0.log"
Sep 07 23:08:43.682795 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:43.682720    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-lv56r_59b03edd-f91a-4d62-af25-1816da1d8641/limitador/0.log"
Sep 07 23:08:43.792729 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:43.792701    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-tqxxf_3789c481-49be-4d95-8dab-83691714b20c/manager/0.log"
Sep 07 23:08:44.234456 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:44.234428    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7sbtl_c4420d24-b59a-4cc3-90b1-f2ed4e48d0be/discovery/0.log"
Sep 07 23:08:44.342443 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:44.342413    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-p9qb4_a26da7d1-cee2-4b67-adcc-8f599c11e65e/kube-auth-proxy/0.log"
Sep 07 23:08:44.875349 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:44.875314    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5789d6f94b-r48qj_2094faa6-7b9d-4f4d-b65a-105f8a62e214/router/0.log"
Sep 07 23:08:45.406066 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:45.406039    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-ft79h_f08fdd25-c8bc-4b51-92e1-f2f2c3c8ec44/main/0.log"
Sep 07 23:08:45.507640 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:45.507615    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-dbptf_87dee3d6-bed9-4e64-b22a-35c607427c3f/main/0.log"
Sep 07 23:08:50.647843 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.647809    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8p8fk/must-gather-2s7hs"]
Sep 07 23:08:50.648281 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648253    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" containerName="revoke-keys"
Sep 07 23:08:50.648281 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648271    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" containerName="revoke-keys"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648308    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648318    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648329    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648340    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648349    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648358    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648370    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648438 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648378    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648820 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648486    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.648820 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648501    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648820 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648511    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e9d457-9dc7-49f3-ad17-3f65e9eb7fa4" containerName="revoke-keys"
Sep 07 23:08:50.648820 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648518    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b048f-c008-42a9-811f-b9de2c5d8735" containerName="istio-proxy"
Sep 07 23:08:50.648820 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.648676    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9649c969-f473-4263-8fb1-acbcd941fe07" containerName="istio-proxy"
Sep 07 23:08:50.651916 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.651895    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.654440 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.654417    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"kube-root-ca.crt\""
Sep 07 23:08:50.655347 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.655328    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8p8fk\"/\"default-dockercfg-ktzj4\""
Sep 07 23:08:50.655421 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.655361    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"openshift-service-ca.crt\""
Sep 07 23:08:50.659177 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.659150    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/must-gather-2s7hs"]
Sep 07 23:08:50.666334 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.666310    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:08:50.762980 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.762943    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-must-gather-output\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.763180 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.762991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxcs\" (UniqueName: \"kubernetes.io/projected/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-kube-api-access-vcxcs\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.864550 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.864509    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxcs\" (UniqueName: \"kubernetes.io/projected/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-kube-api-access-vcxcs\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.864704 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.864601    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-must-gather-output\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.864902 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.864886    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-must-gather-output\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.872525 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.872496    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxcs\" (UniqueName: \"kubernetes.io/projected/02045fbe-15b8-4bfd-b0fd-a489815c6a3a-kube-api-access-vcxcs\") pod \"must-gather-2s7hs\" (UID: \"02045fbe-15b8-4bfd-b0fd-a489815c6a3a\") " pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:50.962710 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:50.962662    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/must-gather-2s7hs"
Sep 07 23:08:51.231563 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:51.231537    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/must-gather-2s7hs"]
Sep 07 23:08:51.233205 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:08:51.233172    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02045fbe_15b8_4bfd_b0fd_a489815c6a3a.slice/crio-5e412d19a3950883750eddc7ac0600304d025f60f4abaf2350fdc73a05d73d7a WatchSource:0}: Error finding container 5e412d19a3950883750eddc7ac0600304d025f60f4abaf2350fdc73a05d73d7a: Status 404 returned error can't find the container with id 5e412d19a3950883750eddc7ac0600304d025f60f4abaf2350fdc73a05d73d7a
Sep 07 23:08:51.235308 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:51.235288    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 23:08:51.281581 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:51.281549    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/must-gather-2s7hs" event={"ID":"02045fbe-15b8-4bfd-b0fd-a489815c6a3a","Type":"ContainerStarted","Data":"5e412d19a3950883750eddc7ac0600304d025f60f4abaf2350fdc73a05d73d7a"}
Sep 07 23:08:52.025291 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:52.025253    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log" limit=1024
Sep 07 23:08:52.291108 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:52.291014    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/must-gather-2s7hs" event={"ID":"02045fbe-15b8-4bfd-b0fd-a489815c6a3a","Type":"ContainerStarted","Data":"67528176dae81ecbbe5656c1c00565fc80578cc8078faf5d0416e3ef14a005ba"}
Sep 07 23:08:52.291108 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:52.291078    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/must-gather-2s7hs" event={"ID":"02045fbe-15b8-4bfd-b0fd-a489815c6a3a","Type":"ContainerStarted","Data":"e82f473b9825fadbdc59ba1b17db475fea16a22661cd47119366d31d529d36a9"}
Sep 07 23:08:52.308036 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:52.307983    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8p8fk/must-gather-2s7hs" podStartSLOduration=1.490598711 podStartE2EDuration="2.307966335s" totalImagesPullingTime="817.367624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:08:52.305776012 +0000 UTC m=+2212.148044325" watchObservedRunningTime="2026-09-07 23:08:52.307966335 +0000 UTC m=+2212.150234603"
Sep 07 23:08:53.545723 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:53.545687    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c7772_c5035533-3c1e-4f5a-a5f2-4d07e4ee8eb2/global-pull-secret-syncer/0.log"
Sep 07 23:08:53.626098 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:53.626056    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9b7wn_9cd8d037-0fbc-43ed-995a-716b71f3edec/konnectivity-agent/0.log"
Sep 07 23:08:53.751460 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:53.751426    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-210.ec2.internal_2f2c00fd5112014ad23844afadb2fd55/haproxy/0.log"
Sep 07 23:08:57.821399 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:57.821367    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-96z4x_5a0c0620-609d-4df0-872a-b383bec31142/manager/0.log"
Sep 07 23:08:57.958589 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:57.958559    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-lv56r_59b03edd-f91a-4d62-af25-1816da1d8641/limitador/0.log"
Sep 07 23:08:58.023985 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:08:58.023937    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-tqxxf_3789c481-49be-4d95-8dab-83691714b20c/manager/0.log"
Sep 07 23:09:00.074522 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.074487    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mgv6d_438607bc-c547-4eac-8656-f01b48c6c696/kube-state-metrics/0.log"
Sep 07 23:09:00.096758 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.096725    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mgv6d_438607bc-c547-4eac-8656-f01b48c6c696/kube-rbac-proxy-main/0.log"
Sep 07 23:09:00.115046 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.115001    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mgv6d_438607bc-c547-4eac-8656-f01b48c6c696/kube-rbac-proxy-self/0.log"
Sep 07 23:09:00.169775 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.169740    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-p7ppd_41af9e32-f8b9-48b0-841f-d64624e079ce/monitoring-plugin/0.log"
Sep 07 23:09:00.329001 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.328922    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5mtm_90a859b0-34bb-4532-afef-3d464da64afb/node-exporter/0.log"
Sep 07 23:09:00.348754 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.348723    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5mtm_90a859b0-34bb-4532-afef-3d464da64afb/kube-rbac-proxy/0.log"
Sep 07 23:09:00.365802 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.365768    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5mtm_90a859b0-34bb-4532-afef-3d464da64afb/init-textfile/0.log"
Sep 07 23:09:00.391206 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.391173    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-8sk99_e85d231a-1af7-42cd-9424-e104025375bf/kube-rbac-proxy-main/0.log"
Sep 07 23:09:00.409547 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.409509    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-8sk99_e85d231a-1af7-42cd-9424-e104025375bf/kube-rbac-proxy-self/0.log"
Sep 07 23:09:00.429054 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.429025    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-8sk99_e85d231a-1af7-42cd-9424-e104025375bf/openshift-state-metrics/0.log"
Sep 07 23:09:00.655913 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.655820    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-2gm7z_ddb208d4-ebf4-4c10-83da-711d9efdd09a/prometheus-operator-admission-webhook/0.log"
Sep 07 23:09:00.751394 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.751366    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/thanos-query/0.log"
Sep 07 23:09:00.772286 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.772258    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/kube-rbac-proxy-web/0.log"
Sep 07 23:09:00.800090 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.800063    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/kube-rbac-proxy/0.log"
Sep 07 23:09:00.815850 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.815812    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/prom-label-proxy/0.log"
Sep 07 23:09:00.850499 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.850464    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/kube-rbac-proxy-rules/0.log"
Sep 07 23:09:00.876146 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:00.874717    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67999497fd-669vw_37a2f84d-6423-4c11-a65c-935d7fceaa1e/kube-rbac-proxy-metrics/0.log"
Sep 07 23:09:01.314883 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:01.314852    2550 scope.go:117] "RemoveContainer" containerID="0171cb6ae7dd9e130fc994d2a5d525b961040c527211593568a091342e70fac6"
Sep 07 23:09:01.330523 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:01.330218    2550 scope.go:117] "RemoveContainer" containerID="89330f5cee28da6764819a1efe8d61996cc18649473a86a6b0a01b563dec4d09"
Sep 07 23:09:02.648224 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:02.648193    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-f7qqt_3af3b600-3fac-4e9f-b577-24d9db07bf35/networking-console-plugin/0.log"
Sep 07 23:09:04.179979 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.179920    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"]
Sep 07 23:09:04.183772 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.183745    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.191018 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.190987    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"]
Sep 07 23:09:04.317426 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.317379    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-podres\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.317615 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.317469    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rppn6\" (UniqueName: \"kubernetes.io/projected/f11a1826-e744-4522-a9bc-e993ce0696e0-kube-api-access-rppn6\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.317615 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.317509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-proc\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.317615 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.317539    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-sys\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.317615 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.317562    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-lib-modules\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418499 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-podres\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rppn6\" (UniqueName: \"kubernetes.io/projected/f11a1826-e744-4522-a9bc-e993ce0696e0-kube-api-access-rppn6\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418567    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-proc\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418597    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-sys\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418622    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-lib-modules\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418633    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-podres\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.418687 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418674    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-proc\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.419015 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-sys\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.419015 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.418762    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a1826-e744-4522-a9bc-e993ce0696e0-lib-modules\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.428867 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.427111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rppn6\" (UniqueName: \"kubernetes.io/projected/f11a1826-e744-4522-a9bc-e993ce0696e0-kube-api-access-rppn6\") pod \"perf-node-gather-daemonset-bhvlc\" (UID: \"f11a1826-e744-4522-a9bc-e993ce0696e0\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.435482 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.435374    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-xltb9_ab1e0fd0-df4c-4232-80ff-b3c222b86ac6/volume-data-source-validator/0.log"
Sep 07 23:09:04.498456 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.498226    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:04.796745 ip-10-0-143-210 kubenswrapper[2550]: W0907 23:09:04.796714    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf11a1826_e744_4522_a9bc_e993ce0696e0.slice/crio-c1761f5e167c47347cd9b040a6e3a5baf463f226c3f6642e844977310a7753c2 WatchSource:0}: Error finding container c1761f5e167c47347cd9b040a6e3a5baf463f226c3f6642e844977310a7753c2: Status 404 returned error can't find the container with id c1761f5e167c47347cd9b040a6e3a5baf463f226c3f6642e844977310a7753c2
Sep 07 23:09:04.801898 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:04.801868    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"]
Sep 07 23:09:05.420709 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.420662    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc" event={"ID":"f11a1826-e744-4522-a9bc-e993ce0696e0","Type":"ContainerStarted","Data":"6af10c28eb2a9efec0376d0abc29fefafde13bbadf7c71a30b443976146c2cf2"}
Sep 07 23:09:05.421237 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.420740    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc" event={"ID":"f11a1826-e744-4522-a9bc-e993ce0696e0","Type":"ContainerStarted","Data":"c1761f5e167c47347cd9b040a6e3a5baf463f226c3f6642e844977310a7753c2"}
Sep 07 23:09:05.421571 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.421541    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:05.592394 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.592366    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-svkjt_b74b79f4-9aac-4c4d-998c-cd09b9e02b53/dns/0.log"
Sep 07 23:09:05.608763 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.608738    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-svkjt_b74b79f4-9aac-4c4d-998c-cd09b9e02b53/kube-rbac-proxy/0.log"
Sep 07 23:09:05.685897 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:05.685821    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz776_b0676c7e-17b2-4596-b67e-9b5d54da77d4/dns-node-resolver/0.log"
Sep 07 23:09:06.240962 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:06.240925    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mwfht_75cb1275-a6c3-40bf-9c65-6a2128a40564/node-ca/0.log"
Sep 07 23:09:07.445106 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:07.445073    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7sbtl_c4420d24-b59a-4cc3-90b1-f2ed4e48d0be/discovery/0.log"
Sep 07 23:09:07.463956 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:07.463924    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-p9qb4_a26da7d1-cee2-4b67-adcc-8f599c11e65e/kube-auth-proxy/0.log"
Sep 07 23:09:07.615215 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:07.615180    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5789d6f94b-r48qj_2094faa6-7b9d-4f4d-b65a-105f8a62e214/router/0.log"
Sep 07 23:09:08.430730 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:08.430694    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k457m_6ec8a08b-16e8-4f9f-89a0-75331e4edb80/serve-healthcheck-canary/0.log"
Sep 07 23:09:08.988210 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:08.988178    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pjg2_9c366661-71e1-467a-84fc-357ab5ee78cb/kube-rbac-proxy/0.log"
Sep 07 23:09:09.005412 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:09.005367    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pjg2_9c366661-71e1-467a-84fc-357ab5ee78cb/exporter/0.log"
Sep 07 23:09:09.022853 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:09.022819    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pjg2_9c366661-71e1-467a-84fc-357ab5ee78cb/extractor/0.log"
Sep 07 23:09:11.087757 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:11.087707    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s2564_429d30f9-8d79-490b-8045-139ee29171a6/manager/0.log"
Sep 07 23:09:11.329151 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:11.329090    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tw2sg_b22bcf05-840c-49b5-bcd5-e896a4419835/server/0.log"
Sep 07 23:09:11.425078 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:11.424989    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtn7q_e1bd5a87-03ac-47d2-b817-a351191470d1/manager/0.log"
Sep 07 23:09:11.446528 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:11.446494    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-btp2q_7da6ed5b-70f7-4893-9061-abf8eb5cde9b/manager/0.log"
Sep 07 23:09:11.465480 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:11.465456    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-btp2q_7da6ed5b-70f7-4893-9061-abf8eb5cde9b/copy-manifests/0.log"
Sep 07 23:09:12.444657 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:12.444625    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc"
Sep 07 23:09:12.463708 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:12.463648    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-bhvlc" podStartSLOduration=8.463627245 podStartE2EDuration="8.463627245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:09:05.437443907 +0000 UTC m=+2225.279712187" watchObservedRunningTime="2026-09-07 23:09:12.463627245 +0000 UTC m=+2232.305895513"
Sep 07 23:09:14.536121 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:14.535959    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7sbtl_c4420d24-b59a-4cc3-90b1-f2ed4e48d0be/discovery/0.log"
Sep 07 23:09:14.546788 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:14.546759    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-p9qb4_a26da7d1-cee2-4b67-adcc-8f599c11e65e/kube-auth-proxy/0.log"
Sep 07 23:09:14.674352 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:14.674268    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5789d6f94b-r48qj_2094faa6-7b9d-4f4d-b65a-105f8a62e214/router/0.log"
Sep 07 23:09:16.745689 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:16.745659    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7sbtl_c4420d24-b59a-4cc3-90b1-f2ed4e48d0be/discovery/0.log"
Sep 07 23:09:16.764210 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:16.764182    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68c66dfc89-p9qb4_a26da7d1-cee2-4b67-adcc-8f599c11e65e/kube-auth-proxy/0.log"
Sep 07 23:09:16.914819 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:16.914737    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5789d6f94b-r48qj_2094faa6-7b9d-4f4d-b65a-105f8a62e214/router/0.log"
Sep 07 23:09:18.430030 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:18.429991    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-t92v8_fdac8917-90cc-4c89-8e2e-3dccfc77a3bc/kube-storage-version-migrator-operator/1.log"
Sep 07 23:09:18.430974 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:18.430947    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-t92v8_fdac8917-90cc-4c89-8e2e-3dccfc77a3bc/kube-storage-version-migrator-operator/0.log"
Sep 07 23:09:19.426250 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.426216    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9nfq6_1a311daa-e34e-4591-a22a-5bf3e012a260/kube-multus/0.log"
Sep 07 23:09:19.570041 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.570016    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/kube-multus-additional-cni-plugins/0.log"
Sep 07 23:09:19.586470 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.586444    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/egress-router-binary-copy/0.log"
Sep 07 23:09:19.602739 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.602718    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/cni-plugins/0.log"
Sep 07 23:09:19.618786 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.618767    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/bond-cni-plugin/0.log"
Sep 07 23:09:19.635876 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.635847    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/routeoverride-cni/0.log"
Sep 07 23:09:19.655298 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.655276    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/whereabouts-cni-bincopy/0.log"
Sep 07 23:09:19.671926 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.671902    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mcb92_257a879d-f99b-4540-9185-342d65e29092/whereabouts-cni/0.log"
Sep 07 23:09:19.942284 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.942255    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hds5x_99661a91-1abb-48c6-99f0-97ee822cab04/network-metrics-daemon/0.log"
Sep 07 23:09:19.956444 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:19.956419    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hds5x_99661a91-1abb-48c6-99f0-97ee822cab04/kube-rbac-proxy/0.log"
Sep 07 23:09:21.158857 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.158827    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/ovn-controller/0.log"
Sep 07 23:09:21.171602 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.171569    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/ovn-acl-logging/0.log"
Sep 07 23:09:21.189963 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.189939    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/kube-rbac-proxy-node/0.log"
Sep 07 23:09:21.205945 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.205922    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 07 23:09:21.221548 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.221529    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/northd/0.log"
Sep 07 23:09:21.237796 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.237778    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/nbdb/0.log"
Sep 07 23:09:21.255742 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.255699    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/sbdb/0.log"
Sep 07 23:09:21.317275 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:21.317249    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kcfs8_6aa259ab-f410-44c8-a25c-e4aa9e05588c/ovnkube-controller/0.log"
Sep 07 23:09:22.308093 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:22.308062    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-p46w7_438316bc-84ec-4b46-827d-ce96b271d110/check-endpoints/0.log"
Sep 07 23:09:22.327123 ip-10-0-143-210 kubenswrapper[2550]: I0907 23:09:22.327097    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-g2k4w_4d2b780d-b803-45c9-b9b7-a6092b85ad76/network-check-target-container/0.log"