Sep 08 16:10:41.294417 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 16:10:41.294429 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 16:10:41.294438 ip-10-0-135-163 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 16:10:41.294681 ip-10-0-135-163 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 16:10:51.404056 ip-10-0-135-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 16:10:51.404069 ip-10-0-135-163 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 79bb2b823c3145e995a783f38508ed1a --
Sep 08 16:12:50.235044 ip-10-0-135-163 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 16:12:50.666078 ip-10-0-135-163 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:50.666078 ip-10-0-135-163 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 16:12:50.666078 ip-10-0-135-163 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:50.666078 ip-10-0-135-163 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 16:12:50.666078 ip-10-0-135-163 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:50.667159 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.667063    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 16:12:50.670053 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670030    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.670053 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670047    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.670053 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670052    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.670053 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670056    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.670053 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670060    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670065    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670070    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670075    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670081    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670085    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670089    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670092    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670096    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670100    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670104    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670108    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670112    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670115    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670119    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670123    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670127    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670131    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670135    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.670335 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670138    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670142    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670146    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670149    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670153    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670157    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670160    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670164    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670168    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670172    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670176    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670180    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670185    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670190    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670194    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670199    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670203    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670207    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670212    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670216    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.671123 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670221    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670229    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670234    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670239    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670244    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670248    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670252    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670256    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670260    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670265    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670269    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670274    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670278    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670282    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670286    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670291    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670294    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670298    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670303    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.671938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670306    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670311    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670315    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670319    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670323    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670327    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670331    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670335    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670341    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670345    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670349    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670354    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670357    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670365    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670370    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670375    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670379    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670384    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670388    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670392    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.672631 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670396    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670400    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670404    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670409    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670414    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670418    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670422    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670427    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670431    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670435    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670439    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670443    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670448    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670452    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670456    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670460    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.670464    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673064    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673077    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673081    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673084    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.673160 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673088    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673091    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673093    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673099    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673109    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673112    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673115    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673118    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673121    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673123    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673126    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673128    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673131    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673134    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673136    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673139    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673141    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673144    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673146    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673149    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.673661 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673152    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673154    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673157    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673159    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673162    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673165    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673168    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673171    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673174    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673177    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673179    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673182    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673184    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673186    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673190    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673192    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673198    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673201    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673204    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.674193 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673207    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673209    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673212    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673215    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673217    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673219    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673222    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673224    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673227    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673229    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673231    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673235    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673240    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673243    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673246    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673249    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673251    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673254    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673256    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.674658 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673259    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673261    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673264    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673266    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673269    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673271    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673274    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673276    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673279    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673282    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673284    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673286    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673289    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673291    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673294    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673296    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673299    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673301    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673303    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673307    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.675255 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673309    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673313    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673316    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673318    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673321    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673323    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673325    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673328    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673330    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673333    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673336    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673338    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673341    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673343    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673346    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673348    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673350    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673730    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673736    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.675733 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673740    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673744    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673748    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673751    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673754    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673756    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673759    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673762    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673783    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673786    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673789    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673792    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673794    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673798    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673801    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673803    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673806    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673809    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673811    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673814    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.676220 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673816    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673819    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673821    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673824    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673827    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673830    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673832    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673834    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673837    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673839    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673842    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673846    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673849    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673853    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673855    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673858    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673861    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673863    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673866    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.676711 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673868    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673871    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673874    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673876    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673879    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673881    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673883    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673887    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673892    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673895    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673897    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673899    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673902    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673904    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673907    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673909    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673911    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673914    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673916    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673919    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.677189 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673921    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673924    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673926    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673929    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673931    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673933    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673936    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673938    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673941    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673943    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673946    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673948    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673951    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673954    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673956    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673959    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673961    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673963    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673966    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673969    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.677700 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673972    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673975    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673977    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673980    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673982    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673985    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673987    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673989    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673992    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673994    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673997    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.673999    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674002    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674004    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674006    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674009    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674011    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674014    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674089    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674120    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 16:12:50.678203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674127    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674131    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674135    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674138    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674142    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674146    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674149    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674152    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674155    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674162    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674165    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674168    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674171    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674174    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674177    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674180    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674185    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674188    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674190    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674194    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674198    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674201    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674204    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 16:12:50.678697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674207    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674211    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674214    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674217    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674220    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674223    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674228    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674232    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674235    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674238    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674241    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674244    2571 flags.go:64] FLAG: --enable-server="true"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674246    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674250    2571 flags.go:64] FLAG: --event-burst="100"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674253    2571 flags.go:64] FLAG: --event-qps="50"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674256    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674259    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674262    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674265    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674268    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674271    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674273    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674276    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674279    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674283    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 16:12:50.679272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674285    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674288    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674291    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674293    2571 flags.go:64] FLAG: --feature-gates=""
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674301    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674304    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674307    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674310    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674312    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674316    2571 flags.go:64] FLAG: --help="false"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674319    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-135-163.ec2.internal"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674321    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674324    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674328    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674333    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674336    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674339    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674342    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674345    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674348    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674351    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674354    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674356    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674359    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 16:12:50.679911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674362    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674365    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674367    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674370    2571 flags.go:64] FLAG: --lock-file=""
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674372    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674375    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674378    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674383    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674387    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674390    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674393    2571 flags.go:64] FLAG: --logging-format="text"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674395    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674399    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674401    2571 flags.go:64] FLAG: --manifest-url=""
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674404    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674408    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674411    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674414    2571 flags.go:64] FLAG: --max-pods="110"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674417    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674420    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674423    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674426    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674430    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674433    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674436    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 16:12:50.680501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674443    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674446    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674449    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674452    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674455    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674461    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674463    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674466    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674469    2571 flags.go:64] FLAG: --port="10250"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674472    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674475    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f53625fe626ff3e"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674478    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674481    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674484    2571 flags.go:64] FLAG: --register-node="true"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674487    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674490    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674494    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674496    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674499    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674502    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674505    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674508    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674511    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674514    2571 flags.go:64] FLAG: --runonce="false"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674516    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 16:12:50.681187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674519    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674522    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674525    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674529    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674532    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674536    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674539    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674542    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674544    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674547    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674550    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674553    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674556    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674558    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674563    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674566    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674569    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674573    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674575    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674578    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674580    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674583    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674586    2571 flags.go:64] FLAG: --v="2"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674590    2571 flags.go:64] FLAG: --version="false"
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674594    2571 flags.go:64] FLAG: --vmodule=""
Sep 08 16:12:50.681810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674597    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.674600    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674714    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674717    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674721    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674724    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674726    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674729    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674732    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674734    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674738    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674740    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674744    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674747    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674749    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674752    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674755    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674757    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674761    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674779    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.682415 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674783    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674786    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674789    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674792    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674794    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674797    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674799    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674802    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674804    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674807    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674809    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674812    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674815    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674817    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674819    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674823    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674827    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674830    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674833    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.682938 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674835    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674838    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674840    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674845    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674849    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674853    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674856    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674859    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674862    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674865    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674867    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674870    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674872    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674875    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674878    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674880    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674883    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674885    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674888    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674890    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.683413 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674893    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674895    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674898    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674900    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674903    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674905    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674909    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674912    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674914    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674917    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674919    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674922    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674924    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674926    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674929    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674933    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674935    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674939    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674941    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674944    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.684003 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674946    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674949    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674952    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674954    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674956    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674959    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674961    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674964    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674967    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674969    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674972    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674974    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674977    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674979    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674981    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674984    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674986    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674989    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674992    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674994    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.684524 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.674997    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.675000    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.675005    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.682786    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.682804    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.684154    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.684162    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684217    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684222    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684225    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684228    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684231    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684234    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684236    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684239    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.685018 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684241    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684245    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684247    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684250    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684253    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684256    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684259    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684261    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684264    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684267    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684269    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684272    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684275    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684279    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684282    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684284    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684287    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684289    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684292    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684294    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.685393 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684297    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684301    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684304    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684307    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684309    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684312    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684314    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684317    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684320    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684322    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684324    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684327    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684330    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684332    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684335    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684338    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684340    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684343    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684345    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684348    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.685885 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684350    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684353    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684355    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684358    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684360    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684364    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684368    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684371    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684374    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684377    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684380    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684383    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684386    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684390    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684393    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684396    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684399    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684402    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684405    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.686358 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684407    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684410    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684412    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684415    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684417    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684419    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684422    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684424    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684427    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684429    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684431    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684434    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684436    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684439    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684442    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684444    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684447    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684449    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684451    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.686834 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684454    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684457    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684459    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684461    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684464    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684466    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684469    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684472    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684475    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684478    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684480    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684483    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684485    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.684491    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684601    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:50.687359 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684606    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684608    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684613    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684616    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684619    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684622    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684625    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684628    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684630    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684633    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684635    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684638    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684641    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684643    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684646    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684648    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684651    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684654    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684656    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:50.687724 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684659    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684661    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684663    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684666    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684668    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684671    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684674    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684676    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684679    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684681    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684684    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684687    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684689    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684691    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684694    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684696    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684699    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684701    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684704    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:50.688213 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684706    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684709    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684711    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684714    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684716    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684718    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684721    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684723    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684726    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684728    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684731    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684733    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684736    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684738    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684741    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684743    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684746    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684748    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684751    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684754    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684756    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:50.688668 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684759    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684762    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684782    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684785    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684788    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684792    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684796    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684798    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684801    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684803    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684806    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684808    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684811    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684813    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684816    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684818    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684821    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684823    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684826    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:50.689274 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684829    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684831    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684834    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684836    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684839    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684841    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684844    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684846    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684849    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684852    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684854    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684857    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684859    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684862    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684864    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684867    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684870    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684872    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684875    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:50.689729 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:50.684877    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.684883    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.684993    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.686970    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.687893    2571 server.go:1019] "Starting client certificate rotation"
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.687997    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:12:50.690221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.688900    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:12:50.690604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.690592    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:12:50.692140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.692123    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:12:50.700391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.700373    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 08 16:12:50.704058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.704039    2571 log.go:25] "Validated CRI v1 image API"
Sep 08 16:12:50.705625 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.705603    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 16:12:50.709399 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.709381    2571 fs.go:135] Filesystem UUIDs: map[6d66335d-c491-4525-8c11-c91f05d72ea3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c8e68c32-8bee-4db1-8cf4-38e46c03ffc7:/dev/nvme0n1p3]
Sep 08 16:12:50.709479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.709400    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 16:12:50.714912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.714808    2571 manager.go:216] Machine: {Timestamp:2026-09-08 16:12:50.714296685 +0000 UTC m=+0.358373369 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102889 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b46e208dd74a6a958bf56881a2a5 SystemUUID:ec22b46e-208d-d74a-6a95-8bf56881a2a5 BootID:79bb2b82-3c31-45e9-95a7-83f38508ed1a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:e3:60:8d:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:e3:60:8d:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:df:67:2d:5a:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 16:12:50.714912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.714908    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 16:12:50.715037 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.715025    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 16:12:50.717880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.717865    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:12:50.718453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.718430    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 16:12:50.718593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.718458    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 16:12:50.719385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.719372    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 16:12:50.719385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.719386    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 16:12:50.719470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.719394    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:12:50.719470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.719408    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:12:50.720169 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.720159    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 16:12:50.724097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724086    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:12:50.724212 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724203    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 16:12:50.724255 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724232    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 16:12:50.724255 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724247    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 16:12:50.724324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724262    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 16:12:50.724324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724271    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 08 16:12:50.724324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724280    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 16:12:50.724648 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724635    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:12:50.724691 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.724656    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:12:50.726906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.726887    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 16:12:50.729201 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.729187    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 16:12:50.729249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.729218    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 16:12:50.730307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730294    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 16:12:50.730359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730315    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 16:12:50.730359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730324    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 16:12:50.730359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730332    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 16:12:50.730359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730345    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 16:12:50.730359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730353    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730372    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730380    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730390    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730398    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730410    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 16:12:50.730517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730418    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 16:12:50.730965 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730954    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 16:12:50.730997 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.730967    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 16:12:50.732659 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.732633    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 16:12:50.732731 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.732680    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 16:12:50.734840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.734826    2571 server.go:1268] "Started kubelet"
Sep 08 16:12:50.734928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.734878    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 16:12:50.734966 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.734895    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 16:12:50.734966 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.734961    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 16:12:50.735633 ip-10-0-135-163 systemd[1]: Started Kubernetes Kubelet.
Sep 08 16:12:50.736858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.736835    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 16:12:50.737481 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.737462    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 16:12:50.738802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.738788    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 16:12:50.741578 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.741558    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 16:12:50.742793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.742759    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 16:12:50.743669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.743645    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 16:12:50.743785 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.743753    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:50.744347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744330    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 16:12:50.744347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744330    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 16:12:50.744458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744359    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 16:12:50.744515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744498    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 16:12:50.744515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744509    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 16:12:50.744604 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.743453    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-163.ec2.internal.18d3647e5409b3e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-163.ec2.internal,UID:ip-10-0-135-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-163.ec2.internal,},FirstTimestamp:2026-09-08 16:12:50.734797798 +0000 UTC m=+0.378874482,LastTimestamp:2026-09-08 16:12:50.734797798 +0000 UTC m=+0.378874482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-163.ec2.internal,}"
Sep 08 16:12:50.744987 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.744967    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gck89"
Sep 08 16:12:50.745147 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745136    2571 factory.go:55] Registering systemd factory
Sep 08 16:12:50.745183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745171    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 08 16:12:50.745388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745374    2571 factory.go:153] Registering CRI-O factory
Sep 08 16:12:50.745425 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745390    2571 factory.go:223] Registration of the crio container factory successfully
Sep 08 16:12:50.745468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745459    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 16:12:50.745502 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745475    2571 factory.go:103] Registering Raw factory
Sep 08 16:12:50.745502 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745484    2571 manager.go:1196] Started watching for new ooms in manager
Sep 08 16:12:50.745828 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.745815    2571 manager.go:349] Starting recovery of all containers
Sep 08 16:12:50.746676 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.746648    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 16:12:50.746813 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.746790    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 16:12:50.753084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.753057    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gck89"
Sep 08 16:12:50.757341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.757328    2571 manager.go:354] Recovery completed
Sep 08 16:12:50.761716 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.761704    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.763736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.763719    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.763836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.763753    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.763836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.763778    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.764306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764294    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 16:12:50.764306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764305    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 16:12:50.764417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764309    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.764417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764322    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:12:50.764970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764954    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.765019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.764987    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.765019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.765000    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.766043 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.765942    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-163.ec2.internal.18d3647e55c346b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-163.ec2.internal,UID:ip-10-0-135-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-163.ec2.internal,},FirstTimestamp:2026-09-08 16:12:50.763736755 +0000 UTC m=+0.407813442,LastTimestamp:2026-09-08 16:12:50.763736755 +0000 UTC m=+0.407813442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-163.ec2.internal,}"
Sep 08 16:12:50.766462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.766450    2571 policy_none.go:49] "None policy: Start"
Sep 08 16:12:50.766514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.766469    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 16:12:50.766514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.766483    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 16:12:50.768587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.768575    2571 policy_none.go:47] "Start"
Sep 08 16:12:50.820376 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820352    2571 manager.go:337] "Starting Device Plugin manager"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.820404    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820419    2571 server.go:87] "Starting device plugin registration server"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820680    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820694    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820800    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820896    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.820908    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.821451    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 16:12:50.829719 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.821502    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:50.880113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.880066    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 16:12:50.881540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.881517    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 16:12:50.881650 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.881580    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 16:12:50.881650 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.881611    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 16:12:50.881722 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.881658    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 16:12:50.883793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.883754    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:50.921896 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.921793    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.922712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.922694    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.922811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.922726    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.922811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.922736    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.922811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.922757    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-163.ec2.internal"
Sep 08 16:12:50.929539 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.929521    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-163.ec2.internal"
Sep 08 16:12:50.929582 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.929549    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-163.ec2.internal\": node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:50.943393 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:50.943369    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:50.982040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.982017    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"]
Sep 08 16:12:50.982124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.982105    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.983687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.983662    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.983804 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.983694    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.983804 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.983704    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.985884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.985872    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.986006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.985993    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:50.986049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986022    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.986563 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986548    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.986617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986572    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.986617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986582    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.986617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986603    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.986718 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986623    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.986718 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.986632    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:50.988651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.988634    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:50.988694 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.988666    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:50.989419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.989404    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:50.989507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.989433    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:50.989507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:50.989446    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:51.005257 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.005237    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-163.ec2.internal\" not found" node="ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.008917 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.008902    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-163.ec2.internal\" not found" node="ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.044303 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.044282    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.045496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.045480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/590fb2a21abc06eff1f2ccb73d30db1a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.ec2.internal\" (UID: \"590fb2a21abc06eff1f2ccb73d30db1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.045550 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.045505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.045550 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.045522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.144743 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.144709    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.145896 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/590fb2a21abc06eff1f2ccb73d30db1a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.ec2.internal\" (UID: \"590fb2a21abc06eff1f2ccb73d30db1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.145945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.145945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.146022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/590fb2a21abc06eff1f2ccb73d30db1a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-163.ec2.internal\" (UID: \"590fb2a21abc06eff1f2ccb73d30db1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.146055 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.146055 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.145988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ad0b8bb437a467220bfb6eda34b9ac5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal\" (UID: \"7ad0b8bb437a467220bfb6eda34b9ac5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.245182 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.245102    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.307345 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.307321    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.312905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.312889    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.345290 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.345267    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.445660 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.445637    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.546038 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.545987    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.645565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.645545    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:51.646483 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.646469    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.687281 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.687256    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 16:12:51.687649 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.687380    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:12:51.687649 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.687399    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:12:51.746628 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:51.746604    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-163.ec2.internal\" not found"
Sep 08 16:12:51.748304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.748288    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:51.758730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.758704    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:07:50 +0000 UTC" deadline="2028-04-26 22:32:50.5120276 +0000 UTC"
Sep 08 16:12:51.758730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.758725    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14310h19m58.753304803s"
Sep 08 16:12:51.844678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.844659    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.858384 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.858367    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:12:51.860531 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.860519    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal"
Sep 08 16:12:51.868368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:51.868353    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:12:52.048156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.048137    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:52.521907 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.521718    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:52.725087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.725060    2571 apiserver.go:52] "Watching apiserver"
Sep 08 16:12:52.735048 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.735028    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 16:12:52.735350 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.735333    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kbt7b","openshift-multus/network-metrics-daemon-7vhdw","openshift-network-operator/iptables-alerter-qxcd9","openshift-ovn-kubernetes/ovnkube-node-lp7kv","kube-system/global-pull-secret-syncer-f6xhh","kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h","openshift-dns/node-resolver-vw7sb","openshift-image-registry/node-ca-ss6b4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal","openshift-multus/multus-pwp4r","openshift-network-diagnostics/network-check-target-gkt2m","kube-system/konnectivity-agent-8cnpg","openshift-cluster-node-tuning-operator/tuned-lwxbc"]
Sep 08 16:12:52.738150 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.738133    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.740033 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.740015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.740276 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.740257    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 16:12:52.740390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.740374    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l4g28\""
Sep 08 16:12:52.740452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.740406    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.740452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.740444    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.741941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.741922    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4lwn4\""
Sep 08 16:12:52.742021 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.741968    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 16:12:52.742175 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.742155    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 16:12:52.742227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.742167    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.743270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.743255    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 16:12:52.743894 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.743880    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.744042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.744028    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 16:12:52.744126 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.744113    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.744197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.744185    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x7mxl\""
Sep 08 16:12:52.744331 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.744320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.746269 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.746253    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.746333 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.746275    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 16:12:52.746333 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.746320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:12:52.747168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747150    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9vp6\""
Sep 08 16:12:52.747237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747168    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.747237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747185    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 16:12:52.747237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747195    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 16:12:52.747237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747206    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 16:12:52.747381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.747281    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.748254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.748242    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.750177 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.750157    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.750252 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.750157    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95v2w\""
Sep 08 16:12:52.750252 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.750238    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.750756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.750742    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.751916 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xgx\" (UniqueName: \"kubernetes.io/projected/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kube-api-access-q2xgx\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.751978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-etc-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.751978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.751978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovn-node-metrics-cert\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.751978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-dbus\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.751983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb88d9e0-bca1-4333-8f8b-f2374b220da5-iptables-alerter-script\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-config\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-env-overrides\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-script-lib\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9jl\" (UniqueName: \"kubernetes.io/projected/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-kube-api-access-wh9jl\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-log-socket\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-etc-selinux\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752121    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kpw\" (UniqueName: \"kubernetes.io/projected/fb88d9e0-bca1-4333-8f8b-f2374b220da5-kube-api-access-z4kpw\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.752132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752134    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-systemd-units\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-bin\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-sys-fs\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752195    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb88d9e0-bca1-4333-8f8b-f2374b220da5-host-slash\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-kubelet\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-node-log\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752280    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ced490-a938-46a8-856b-07376d703c79-agent-certs\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-slash\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-var-lib-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-netd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-kubelet-config\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-netns\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.752494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-systemd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-registration-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-device-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ced490-a938-46a8-856b-07376d703c79-konnectivity-ca\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-ovn\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-socket-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752881    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b88bx\""
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752910    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.753020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.752985    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.753328 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.753129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.754268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.754250    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:12:52.754946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.754924    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zt44w\""
Sep 08 16:12:52.755049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755033    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 16:12:52.755127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.755321 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755309    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.755369 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.755316    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:12:52.755493 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755476    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 16:12:52.755559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755510    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.755559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.755533    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 16:12:52.759203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.758554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.760580 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.760555    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.760858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.760839    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 16:12:52.761019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.761000    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6ksrb\""
Sep 08 16:12:52.761099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.761035    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 16:12:52.763070 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.763055    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nznrf\""
Sep 08 16:12:52.763142 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.763074    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 16:12:52.763479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.763466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:52.763525 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.763508    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:12:52.773537 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.773494    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hvhzt"
Sep 08 16:12:52.781385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.781370    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hvhzt"
Sep 08 16:12:52.825403 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:52.825383    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad0b8bb437a467220bfb6eda34b9ac5.slice/crio-8e408ef8e9c3a8e241f395d44cde8496008e7fa077d2a8bdb34d40cfb714f03d WatchSource:0}: Error finding container 8e408ef8e9c3a8e241f395d44cde8496008e7fa077d2a8bdb34d40cfb714f03d: Status 404 returned error can't find the container with id 8e408ef8e9c3a8e241f395d44cde8496008e7fa077d2a8bdb34d40cfb714f03d
Sep 08 16:12:52.826073 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:52.826055    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590fb2a21abc06eff1f2ccb73d30db1a.slice/crio-1a7692192bc9b3f599aeab39f5287c943d024c4ea2ed751c58e86480fce2c26e WatchSource:0}: Error finding container 1a7692192bc9b3f599aeab39f5287c943d024c4ea2ed751c58e86480fce2c26e: Status 404 returned error can't find the container with id 1a7692192bc9b3f599aeab39f5287c943d024c4ea2ed751c58e86480fce2c26e
Sep 08 16:12:52.829763 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.829740    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:12:52.845099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.845086    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 16:12:52.852967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.852952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-kubelet\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.852973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-hostroot\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.852989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-conf\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.853019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-lib-modules\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.853151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-registration-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.853151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ced490-a938-46a8-856b-07376d703c79-konnectivity-ca\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.853151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-tuned\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.853151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.853151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-registration-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8hw\" (UniqueName: \"kubernetes.io/projected/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-kube-api-access-zd8hw\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-k8s-cni-cncf-io\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853244    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-system-cni-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovn-node-metrics-cert\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-bin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-sys\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb88d9e0-bca1-4333-8f8b-f2374b220da5-iptables-alerter-script\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ced490-a938-46a8-856b-07376d703c79-konnectivity-ca\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-script-lib\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-netns\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-etc-kubernetes\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-log-socket\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-systemd-units\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-daemon-config\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853643    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 16:12:52.853674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-log-socket\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853680    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-systemd-units\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.853735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrrc\" (UniqueName: \"kubernetes.io/projected/ecd65ad8-a513-44bf-9bdc-986b9e57190e-kube-api-access-tnrrc\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-bin\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.853847    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-slash\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-var-lib-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854201    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-socket-dir-parent\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.854212    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:53.354190296 +0000 UTC m=+2.998266982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb88d9e0-bca1-4333-8f8b-f2374b220da5-iptables-alerter-script\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-script-lib\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-bin\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-slash\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysconfig\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854381    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-var-lib-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854417    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7qt\" (UniqueName: \"kubernetes.io/projected/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-kube-api-access-sg7qt\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-netns\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3586bf-382f-44e6-90bd-7dd42f1489f8-tmp-dir\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-multus\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-netns\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-device-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-device-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.854882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-ovn\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854903    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-system-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-kubernetes\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854970    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-socket-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.854972    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-ovn\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xgx\" (UniqueName: \"kubernetes.io/projected/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kube-api-access-q2xgx\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-socket-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855066    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-etc-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxv2\" (UniqueName: \"kubernetes.io/projected/c0fdae80-64a3-45e9-a522-b351f3d85967-kube-api-access-7bxv2\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-var-lib-kubelet\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cnibin\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-dbus\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-host\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-etc-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-config\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-env-overrides\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-dbus\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9jl\" (UniqueName: \"kubernetes.io/projected/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-kube-api-access-wh9jl\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-openvswitch\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.855723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855699    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-cni-binary-copy\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-modprobe-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-run\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-etc-selinux\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kpw\" (UniqueName: \"kubernetes.io/projected/fb88d9e0-bca1-4333-8f8b-f2374b220da5-kube-api-access-z4kpw\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngq6\" (UniqueName: \"kubernetes.io/projected/0e3586bf-382f-44e6-90bd-7dd42f1489f8-kube-api-access-5ngq6\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-systemd\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-env-overrides\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.855995    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-sys-fs\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-etc-selinux\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb88d9e0-bca1-4333-8f8b-f2374b220da5-host-slash\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-kubelet\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb88d9e0-bca1-4333-8f8b-f2374b220da5-host-slash\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856102    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-sys-fs\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovnkube-config\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-node-log\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.856317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-node-log\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856193    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-kubelet\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-cnibin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-conf-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-multus-certs\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92hw\" (UniqueName: \"kubernetes.io/projected/0ac91bfe-f454-49ba-97c9-d52c97c198c7-kube-api-access-t92hw\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ced490-a938-46a8-856b-07376d703c79-agent-certs\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-netd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-kubelet-config\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-tmp\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-host\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e0303ba-eecf-48b6-887a-87fcf3f78535-kubelet-config\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-serviceca\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-systemd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e3586bf-382f-44e6-90bd-7dd42f1489f8-hosts-file\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.857821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856809    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.857821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-run-systemd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.856913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.857821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.857060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-host-cni-netd\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.858081 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.858063    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-ovn-node-metrics-cert\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.858904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.858887    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ced490-a938-46a8-856b-07376d703c79-agent-certs\") pod \"konnectivity-agent-8cnpg\" (UID: \"00ced490-a938-46a8-856b-07376d703c79\") " pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:52.862447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.862429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xgx\" (UniqueName: \"kubernetes.io/projected/e3e11b0e-11fc-438b-ae4b-867e1a16e713-kube-api-access-q2xgx\") pod \"aws-ebs-csi-driver-node-slt4h\" (UID: \"e3e11b0e-11fc-438b-ae4b-867e1a16e713\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:52.862536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.862463    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9jl\" (UniqueName: \"kubernetes.io/projected/1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5-kube-api-access-wh9jl\") pod \"ovnkube-node-lp7kv\" (UID: \"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:52.862617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.862601    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kpw\" (UniqueName: \"kubernetes.io/projected/fb88d9e0-bca1-4333-8f8b-f2374b220da5-kube-api-access-z4kpw\") pod \"iptables-alerter-qxcd9\" (UID: \"fb88d9e0-bca1-4333-8f8b-f2374b220da5\") " pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:52.885652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.885623    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal" event={"ID":"7ad0b8bb437a467220bfb6eda34b9ac5","Type":"ContainerStarted","Data":"8e408ef8e9c3a8e241f395d44cde8496008e7fa077d2a8bdb34d40cfb714f03d"}
Sep 08 16:12:52.886567 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.886546    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal" event={"ID":"590fb2a21abc06eff1f2ccb73d30db1a","Type":"ContainerStarted","Data":"1a7692192bc9b3f599aeab39f5287c943d024c4ea2ed751c58e86480fce2c26e"}
Sep 08 16:12:52.956956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.956933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-cnibin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.956960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-conf-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.956980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-multus-certs\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92hw\" (UniqueName: \"kubernetes.io/projected/0ac91bfe-f454-49ba-97c9-d52c97c198c7-kube-api-access-t92hw\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.957045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957023    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-tmp\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.957045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-conf-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957045    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-multus-certs\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-host\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957041    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-cnibin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-serviceca\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-host\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e3586bf-382f-44e6-90bd-7dd42f1489f8-hosts-file\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-kubelet\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-hostroot\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957171    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-conf\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957215    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-kubelet\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957228    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-hostroot\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957214    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-lib-modules\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e3586bf-382f-44e6-90bd-7dd42f1489f8-hosts-file\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-tuned\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.957277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957299    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-lib-modules\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957318    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-conf\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8hw\" (UniqueName: \"kubernetes.io/projected/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-kube-api-access-zd8hw\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957366    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-k8s-cni-cncf-io\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-system-cni-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-bin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-sys\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-netns\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957485    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-system-cni-dir\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-etc-kubernetes\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957529    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-k8s-cni-cncf-io\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-serviceca\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957599    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-bin\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.957627    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:52.958023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-run-netns\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.957670    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:12:53.457656161 +0000 UTC m=+3.101732841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-sys\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-daemon-config\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957712    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-etc-kubernetes\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957727    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrrc\" (UniqueName: \"kubernetes.io/projected/ecd65ad8-a513-44bf-9bdc-986b9e57190e-kube-api-access-tnrrc\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957755    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-socket-dir-parent\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysconfig\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957863    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7qt\" (UniqueName: \"kubernetes.io/projected/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-kube-api-access-sg7qt\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysctl-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3586bf-382f-44e6-90bd-7dd42f1489f8-tmp-dir\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-multus\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-sysconfig\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-system-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.958824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-kubernetes\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.957985    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-socket-dir-parent\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxv2\" (UniqueName: \"kubernetes.io/projected/c0fdae80-64a3-45e9-a522-b351f3d85967-kube-api-access-7bxv2\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-host-var-lib-cni-multus\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958120    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-system-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-var-lib-kubelet\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-kubernetes\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3586bf-382f-44e6-90bd-7dd42f1489f8-tmp-dir\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958213    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-daemon-config\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cnibin\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-var-lib-kubelet\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-host\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958268    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-cnibin\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958294    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-host\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-cni-binary-copy\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-modprobe-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-run\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958408    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0fdae80-64a3-45e9-a522-b351f3d85967-multus-cni-dir\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngq6\" (UniqueName: \"kubernetes.io/projected/0e3586bf-382f-44e6-90bd-7dd42f1489f8-kube-api-access-5ngq6\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-systemd\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-systemd\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-modprobe-d\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ac91bfe-f454-49ba-97c9-d52c97c198c7-run\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958715    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.958817    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0fdae80-64a3-45e9-a522-b351f3d85967-cni-binary-copy\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.959131    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-tmp\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.959728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.959654    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ac91bfe-f454-49ba-97c9-d52c97c198c7-etc-tuned\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.963662 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.963647    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:52.963749 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.963669    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:52.963749 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.963705    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:52.963868 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:52.963750    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:53.463740188 +0000 UTC m=+3.107816868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:52.966273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.966252    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8hw\" (UniqueName: \"kubernetes.io/projected/b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574-kube-api-access-zd8hw\") pod \"node-ca-ss6b4\" (UID: \"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574\") " pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:52.966371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.966282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrrc\" (UniqueName: \"kubernetes.io/projected/ecd65ad8-a513-44bf-9bdc-986b9e57190e-kube-api-access-tnrrc\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:52.966371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.966312    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92hw\" (UniqueName: \"kubernetes.io/projected/0ac91bfe-f454-49ba-97c9-d52c97c198c7-kube-api-access-t92hw\") pod \"tuned-lwxbc\" (UID: \"0ac91bfe-f454-49ba-97c9-d52c97c198c7\") " pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:52.966470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.966390    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxv2\" (UniqueName: \"kubernetes.io/projected/c0fdae80-64a3-45e9-a522-b351f3d85967-kube-api-access-7bxv2\") pod \"multus-pwp4r\" (UID: \"c0fdae80-64a3-45e9-a522-b351f3d85967\") " pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:52.967085 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.967070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngq6\" (UniqueName: \"kubernetes.io/projected/0e3586bf-382f-44e6-90bd-7dd42f1489f8-kube-api-access-5ngq6\") pod \"node-resolver-vw7sb\" (UID: \"0e3586bf-382f-44e6-90bd-7dd42f1489f8\") " pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:52.967183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:52.967167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7qt\" (UniqueName: \"kubernetes.io/projected/434f72bb-ef9a-4917-8bf1-bf0ff3e9258b-kube-api-access-sg7qt\") pod \"multus-additional-cni-plugins-kbt7b\" (UID: \"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b\") " pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:53.075587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.075568    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h"
Sep 08 16:12:53.086163 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.086142    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e11b0e_11fc_438b_ae4b_867e1a16e713.slice/crio-b710a73fa0dde8e48d0f3825f97e165ca9a6e53075f0cc36b607f1074ab514a4 WatchSource:0}: Error finding container b710a73fa0dde8e48d0f3825f97e165ca9a6e53075f0cc36b607f1074ab514a4: Status 404 returned error can't find the container with id b710a73fa0dde8e48d0f3825f97e165ca9a6e53075f0cc36b607f1074ab514a4
Sep 08 16:12:53.092033 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.092013    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:12:53.099005 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.098988    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ced490_a938_46a8_856b_07376d703c79.slice/crio-9e849117a651a8fadaa796dad23a82b1cda059f370aeb12038eb42a8304e4761 WatchSource:0}: Error finding container 9e849117a651a8fadaa796dad23a82b1cda059f370aeb12038eb42a8304e4761: Status 404 returned error can't find the container with id 9e849117a651a8fadaa796dad23a82b1cda059f370aeb12038eb42a8304e4761
Sep 08 16:12:53.107137 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.107122    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qxcd9"
Sep 08 16:12:53.114137 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.114113    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb88d9e0_bca1_4333_8f8b_f2374b220da5.slice/crio-d613f081d97e1ad413b0a1d6314c8182efb31a4eb11f67e9a71e17cba4018ffe WatchSource:0}: Error finding container d613f081d97e1ad413b0a1d6314c8182efb31a4eb11f67e9a71e17cba4018ffe: Status 404 returned error can't find the container with id d613f081d97e1ad413b0a1d6314c8182efb31a4eb11f67e9a71e17cba4018ffe
Sep 08 16:12:53.124703 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.124673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:12:53.131382 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.131365    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c20d2ca_5bf2_4624_bb7f_5cd35eb4a9f5.slice/crio-aaffe210520e721c1c1a8d000046d5adcf96ef218e53e30f52e1cead89cbfb72 WatchSource:0}: Error finding container aaffe210520e721c1c1a8d000046d5adcf96ef218e53e30f52e1cead89cbfb72: Status 404 returned error can't find the container with id aaffe210520e721c1c1a8d000046d5adcf96ef218e53e30f52e1cead89cbfb72
Sep 08 16:12:53.134126 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.134111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwxbc"
Sep 08 16:12:53.138957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.138944    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ss6b4"
Sep 08 16:12:53.140975 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.140960    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac91bfe_f454_49ba_97c9_d52c97c198c7.slice/crio-02d66bee29367a7a3ef19d70e471bc898ba75394542b6c37a5031eac6229c56a WatchSource:0}: Error finding container 02d66bee29367a7a3ef19d70e471bc898ba75394542b6c37a5031eac6229c56a: Status 404 returned error can't find the container with id 02d66bee29367a7a3ef19d70e471bc898ba75394542b6c37a5031eac6229c56a
Sep 08 16:12:53.143829 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.143751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbt7b"
Sep 08 16:12:53.146583 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.146565    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40d2ffe_0c82_4b9c_9cd0_26c8b17a2574.slice/crio-526ecf003f8e70686095989171485ad6f4b1d8e5b864b686ce5c5a4c2cbecfea WatchSource:0}: Error finding container 526ecf003f8e70686095989171485ad6f4b1d8e5b864b686ce5c5a4c2cbecfea: Status 404 returned error can't find the container with id 526ecf003f8e70686095989171485ad6f4b1d8e5b864b686ce5c5a4c2cbecfea
Sep 08 16:12:53.148871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.148853    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vw7sb"
Sep 08 16:12:53.152004 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.151989    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434f72bb_ef9a_4917_8bf1_bf0ff3e9258b.slice/crio-ddf5e130c47eb6b5557b16a628534e9462e84e9f0c02b74853c96ded5fc89329 WatchSource:0}: Error finding container ddf5e130c47eb6b5557b16a628534e9462e84e9f0c02b74853c96ded5fc89329: Status 404 returned error can't find the container with id ddf5e130c47eb6b5557b16a628534e9462e84e9f0c02b74853c96ded5fc89329
Sep 08 16:12:53.152945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.152924    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pwp4r"
Sep 08 16:12:53.158621 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.158601    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3586bf_382f_44e6_90bd_7dd42f1489f8.slice/crio-5a8bb564444e1bea50a01f02b5c558342853bab56141f3030cef0ba73e6401c4 WatchSource:0}: Error finding container 5a8bb564444e1bea50a01f02b5c558342853bab56141f3030cef0ba73e6401c4: Status 404 returned error can't find the container with id 5a8bb564444e1bea50a01f02b5c558342853bab56141f3030cef0ba73e6401c4
Sep 08 16:12:53.162257 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:12:53.162239    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fdae80_64a3_45e9_a522_b351f3d85967.slice/crio-53f03f9948467d8946239f6c61e876fbfd419752f7046159288ceb869b507f75 WatchSource:0}: Error finding container 53f03f9948467d8946239f6c61e876fbfd419752f7046159288ceb869b507f75: Status 404 returned error can't find the container with id 53f03f9948467d8946239f6c61e876fbfd419752f7046159288ceb869b507f75
Sep 08 16:12:53.361114 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.361038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:53.361244 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.361172    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:53.361244 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.361241    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:54.361228251 +0000 UTC m=+4.005304929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:53.462002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.461964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:53.462144 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.462128    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:53.462233 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.462217    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:12:54.462181748 +0000 UTC m=+4.106258429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:53.563582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.562962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:53.563582 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.563153    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:53.563582 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.563171    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:53.563582 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.563183    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:53.563582 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.563258    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:54.56323989 +0000 UTC m=+4.207316563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:53.784700 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.784613    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:52 +0000 UTC" deadline="2028-03-22 20:50:41.395031863 +0000 UTC"
Sep 08 16:12:53.784700 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.784653    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13468h37m47.610383949s"
Sep 08 16:12:53.884992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.884959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:53.885160 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.885084    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:12:53.885220 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.885162    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:53.885266 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:53.885234    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:12:53.910982 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.910949    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerStarted","Data":"ddf5e130c47eb6b5557b16a628534e9462e84e9f0c02b74853c96ded5fc89329"}
Sep 08 16:12:53.923395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.923362    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwxbc" event={"ID":"0ac91bfe-f454-49ba-97c9-d52c97c198c7","Type":"ContainerStarted","Data":"02d66bee29367a7a3ef19d70e471bc898ba75394542b6c37a5031eac6229c56a"}
Sep 08 16:12:53.930570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.930540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qxcd9" event={"ID":"fb88d9e0-bca1-4333-8f8b-f2374b220da5","Type":"ContainerStarted","Data":"d613f081d97e1ad413b0a1d6314c8182efb31a4eb11f67e9a71e17cba4018ffe"}
Sep 08 16:12:53.949624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.949590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8cnpg" event={"ID":"00ced490-a938-46a8-856b-07376d703c79","Type":"ContainerStarted","Data":"9e849117a651a8fadaa796dad23a82b1cda059f370aeb12038eb42a8304e4761"}
Sep 08 16:12:53.961135 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.961092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pwp4r" event={"ID":"c0fdae80-64a3-45e9-a522-b351f3d85967","Type":"ContainerStarted","Data":"53f03f9948467d8946239f6c61e876fbfd419752f7046159288ceb869b507f75"}
Sep 08 16:12:53.966589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.966371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vw7sb" event={"ID":"0e3586bf-382f-44e6-90bd-7dd42f1489f8","Type":"ContainerStarted","Data":"5a8bb564444e1bea50a01f02b5c558342853bab56141f3030cef0ba73e6401c4"}
Sep 08 16:12:53.984357 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.984307    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ss6b4" event={"ID":"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574","Type":"ContainerStarted","Data":"526ecf003f8e70686095989171485ad6f4b1d8e5b864b686ce5c5a4c2cbecfea"}
Sep 08 16:12:53.989470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.989441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"aaffe210520e721c1c1a8d000046d5adcf96ef218e53e30f52e1cead89cbfb72"}
Sep 08 16:12:53.996377 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:53.996350    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h" event={"ID":"e3e11b0e-11fc-438b-ae4b-867e1a16e713","Type":"ContainerStarted","Data":"b710a73fa0dde8e48d0f3825f97e165ca9a6e53075f0cc36b607f1074ab514a4"}
Sep 08 16:12:54.368961 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.368925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:54.369121 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.369068    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:54.369121 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.369120    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.369104714 +0000 UTC m=+6.013181386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:54.471035 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.469984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:54.471035 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.470139    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:54.471035 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.470199    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.470181353 +0000 UTC m=+6.114258024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:54.571080 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.571008    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:54.571220 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.571150    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:54.571220 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.571167    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:54.571220 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.571178    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:54.571377 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.571230    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.571213882 +0000 UTC m=+6.215290553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:54.591129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.590940    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:54.785046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.784986    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:52 +0000 UTC" deadline="2028-03-28 18:16:29.744446538 +0000 UTC"
Sep 08 16:12:54.785046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.785039    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13610h3m34.959411355s"
Sep 08 16:12:54.884843 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:54.884754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:54.884999 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:54.884900    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:12:55.882355 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:55.881778    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:55.882355 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:55.881891    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:12:55.882355 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:55.881957    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:55.882355 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:55.882026    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:12:56.384235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:56.383639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:56.384235 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.383876    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:56.384235 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.383947    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:00.383923421 +0000 UTC m=+10.028000091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:56.484560 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:56.484520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:56.484733 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.484718    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:56.484961 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.484793    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:13:00.484756056 +0000 UTC m=+10.128832724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:56.585078 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:56.585038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:56.585272 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.585215    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:56.585272 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.585237    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:56.585272 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.585249    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:56.585443 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.585304    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:00.585287467 +0000 UTC m=+10.229364141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:56.882295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:56.881880    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:56.882295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:56.882015    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:12:57.882426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:57.882389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:57.882625 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:57.882388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:57.882625 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:57.882526    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:12:57.882625 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:57.882611    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:12:58.882641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:58.882596    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:12:58.883055 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:58.882748    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:12:59.882361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:59.882276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:12:59.882529 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:59.882454    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:12:59.882869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:12:59.882854    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:12:59.883139 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:12:59.882924    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:00.419634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:00.419601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:00.419850 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.419725    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:00.419850 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.419818    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:08.419804847 +0000 UTC m=+18.063881516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:00.520867 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:00.520822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:00.521034 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.521003    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:00.521117 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.521065    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:13:08.52104711 +0000 UTC m=+18.165123797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:00.621426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:00.621386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:00.621658 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.621630    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:13:00.621658 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.621664    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:13:00.621875 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.621678    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:00.621875 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.621741    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:08.621721258 +0000 UTC m=+18.265797946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:00.885654 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:00.885622    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:00.886100 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:00.885738    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:01.882663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:01.882623    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:01.882663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:01.882648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:01.882940 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:01.882747    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:01.882940 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:01.882871    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:02.882120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:02.882080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:02.882599 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:02.882231    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:03.882561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:03.882522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:03.882561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:03.882568    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:03.883059 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:03.882713    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:03.883059 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:03.882853    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:04.882723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:04.882691    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:04.883196 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:04.882851    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:05.882029 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:05.881992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:05.882213 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:05.881992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:05.882213 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:05.882123    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:05.882213 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:05.882193    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:06.882295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:06.882261    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:06.882703 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:06.882404    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:07.881818 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:07.881793    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:07.881922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:07.881796    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:07.881922 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:07.881886    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:07.881989 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:07.881961    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:08.473956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:08.473915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:08.474483 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.474095    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:08.474483 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.474180    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:24.474157986 +0000 UTC m=+34.118234673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:08.574294 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:08.574257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:08.574445 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.574408    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:08.574513 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.574479    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:13:24.57446132 +0000 UTC m=+34.218537992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:08.675563 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:08.675525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:08.675743 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.675666    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:13:08.675743 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.675683    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:13:08.675743 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.675695    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:08.675887 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.675756    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:24.67573648 +0000 UTC m=+34.319813155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:08.882132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:08.882104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:08.882372 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:08.882234    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:09.882520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:09.882485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:09.882520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:09.882522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:09.883035 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:09.882590    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:09.883035 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:09.882699    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:10.882964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:10.882815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:10.882964 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:10.882924    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:11.882214 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:11.882092    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:11.882295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:11.882096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:11.882295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:11.882284    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:11.882360 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:11.882348    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:12.045167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.045133    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwxbc" event={"ID":"0ac91bfe-f454-49ba-97c9-d52c97c198c7","Type":"ContainerStarted","Data":"2ab9846c778250c3efdd8cf5aedd67f05f62b2dfa0227f9c6374c464ea309afb"}
Sep 08 16:13:12.046586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.046565    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal" event={"ID":"590fb2a21abc06eff1f2ccb73d30db1a","Type":"ContainerStarted","Data":"2c396f3790d23d3795fa67714aebd26ed6d07b1280f10698205b5da50ccd589d"}
Sep 08 16:13:12.047909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.047887    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"ffa8b6f192ac6873884f8c036bf2c0b25444408891c4ca8de2ac9c72327fb107"}
Sep 08 16:13:12.066087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.066044    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lwxbc" podStartSLOduration=3.427098963 podStartE2EDuration="22.066033159s" totalImagesPullingTime="18.638934196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:12.065924083 +0000 UTC m=+21.710000774" watchObservedRunningTime="2026-09-08 16:13:12.066033159 +0000 UTC m=+21.710109851"
Sep 08 16:13:12.078033 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.077997    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-163.ec2.internal" podStartSLOduration=21.07798479 podStartE2EDuration="21.07798479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:12.077903136 +0000 UTC m=+21.721979829" watchObservedRunningTime="2026-09-08 16:13:12.07798479 +0000 UTC m=+21.722061482"
Sep 08 16:13:12.882560 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:12.882301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:12.882693 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:12.882608    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:13.051938 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.051905    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pwp4r" event={"ID":"c0fdae80-64a3-45e9-a522-b351f3d85967","Type":"ContainerStarted","Data":"ea3e2618be5310871e0dd98132173a7f006dc2ff0e8c95d1aeae99893100897d"}
Sep 08 16:13:13.053391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.053367    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vw7sb" event={"ID":"0e3586bf-382f-44e6-90bd-7dd42f1489f8","Type":"ContainerStarted","Data":"7782a3d840d9eec58af1cf7f36bc226509da37468fdf2503e452b3b8867b9de8"}
Sep 08 16:13:13.054856 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.054829    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ss6b4" event={"ID":"b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574","Type":"ContainerStarted","Data":"666b958da8773b04169ef4c0e74a3bdb8e92c0ad2cc8c4e5be32d924e96a3819"}
Sep 08 16:13:13.058314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:13:13.058745 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058706    2571 generic.go:356] "Generic (PLEG): container finished" podID="1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5" containerID="3f0604315c8b20eedd00f3aef34151fde56bfe8454d3cd2e75480ed5a7524e98" exitCode=1
Sep 08 16:13:13.058834 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058749    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"2f61eb2e7a2c21ae7e02463e9bb788a016a928a193e46dc75b23bdb22b403223"}
Sep 08 16:13:13.058834 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058821    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"87ba8099ff749f94407749f068936c3e9f71e297c7c5509e52de0f533ef646b9"}
Sep 08 16:13:13.058951 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058868    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"762f280c3f75a00079421d27b2cca5c4b74ad40e25f4a283aa4bd8378f7b5f05"}
Sep 08 16:13:13.058951 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"b14c251268c139d1b52abcc013eba33af2ee467d50eb7268ebf785ba41918db3"}
Sep 08 16:13:13.059022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.058964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerDied","Data":"3f0604315c8b20eedd00f3aef34151fde56bfe8454d3cd2e75480ed5a7524e98"}
Sep 08 16:13:13.060381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.060363    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h" event={"ID":"e3e11b0e-11fc-438b-ae4b-867e1a16e713","Type":"ContainerStarted","Data":"5553e9f10c261ce7e2651cd9d098a6c3f556db293a9fef2d333078e84bc8d183"}
Sep 08 16:13:13.061920 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.061899    2571 generic.go:356] "Generic (PLEG): container finished" podID="7ad0b8bb437a467220bfb6eda34b9ac5" containerID="e0121f6fd565b134c68317bdf015e7cdb5f7c4ed9b4e908fdc5739cbdf848180" exitCode=0
Sep 08 16:13:13.062006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.061928    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal" event={"ID":"7ad0b8bb437a467220bfb6eda34b9ac5","Type":"ContainerDied","Data":"e0121f6fd565b134c68317bdf015e7cdb5f7c4ed9b4e908fdc5739cbdf848180"}
Sep 08 16:13:13.063528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.063509    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="66c3c18d2d851d531f4d5a95d45aa82caf177de8411e1afcebcf7cdc31145978" exitCode=0
Sep 08 16:13:13.063618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.063572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"66c3c18d2d851d531f4d5a95d45aa82caf177de8411e1afcebcf7cdc31145978"}
Sep 08 16:13:13.065171 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.065137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8cnpg" event={"ID":"00ced490-a938-46a8-856b-07376d703c79","Type":"ContainerStarted","Data":"913fd536231e0de249d97cc486e331f720a22fc690e09f2e9f13ecb466992d7d"}
Sep 08 16:13:13.068199 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.068122    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pwp4r" podStartSLOduration=4.101680485 podStartE2EDuration="23.068109448s" totalImagesPullingTime="18.966428963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:13.067253921 +0000 UTC m=+22.711330614" watchObservedRunningTime="2026-09-08 16:13:13.068109448 +0000 UTC m=+22.712186137"
Sep 08 16:13:13.080439 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.080398    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vw7sb" podStartSLOduration=4.458627038 podStartE2EDuration="23.08038594s" totalImagesPullingTime="18.621758902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:13.080359778 +0000 UTC m=+22.724436473" watchObservedRunningTime="2026-09-08 16:13:13.08038594 +0000 UTC m=+22.724462629"
Sep 08 16:13:13.107536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.107497    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8cnpg" podStartSLOduration=8.191976001 podStartE2EDuration="23.10748465s" totalImagesPullingTime="14.915508649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:13.107018605 +0000 UTC m=+22.751095298" watchObservedRunningTime="2026-09-08 16:13:13.10748465 +0000 UTC m=+22.751561343"
Sep 08 16:13:13.122479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.122437    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ss6b4" podStartSLOduration=4.5083000680000005 podStartE2EDuration="23.122424385s" totalImagesPullingTime="18.614124317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:13.121992581 +0000 UTC m=+22.766069274" watchObservedRunningTime="2026-09-08 16:13:13.122424385 +0000 UTC m=+22.766501090"
Sep 08 16:13:13.329960 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.329807    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 16:13:13.832402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.832254    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:13:13.329956599Z","UUID":"0a42c36b-4d15-4554-b9f1-43cc24b98327","Handler":null,"Name":"","Endpoint":""}
Sep 08 16:13:13.834023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.834001    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 16:13:13.834155 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.834034    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 16:13:13.882333 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.882310    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:13.882476 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:13.882403    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:13.882705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:13.882688    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:13.882845 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:13.882814    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:14.069707 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.069671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h" event={"ID":"e3e11b0e-11fc-438b-ae4b-867e1a16e713","Type":"ContainerStarted","Data":"e64755a36a4f40cd6976825ab466a080ddf3ed51984640b96fc1c4422c3a8ab4"}
Sep 08 16:13:14.072164 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.072137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal" event={"ID":"7ad0b8bb437a467220bfb6eda34b9ac5","Type":"ContainerStarted","Data":"29e0c8f381021d8eda79d6ee47daba2bc926f397d6897b2202ae7d6128e7d79a"}
Sep 08 16:13:14.074870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.074833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qxcd9" event={"ID":"fb88d9e0-bca1-4333-8f8b-f2374b220da5","Type":"ContainerStarted","Data":"a0a5df2395a307627ea1d6c14907cc3217603582509c42f52e5aa5bc16dd81c1"}
Sep 08 16:13:14.086593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.086510    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-163.ec2.internal" podStartSLOduration=23.086496022 podStartE2EDuration="23.086496022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:14.086298816 +0000 UTC m=+23.730375511" watchObservedRunningTime="2026-09-08 16:13:14.086496022 +0000 UTC m=+23.730572715"
Sep 08 16:13:14.882324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.882290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:14.882492 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:14.882422    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:14.977816 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.977787    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:13:14.978372 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.978357    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:13:14.991905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:14.991852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qxcd9" podStartSLOduration=10.09151618 podStartE2EDuration="24.991831169s" totalImagesPullingTime="14.900314989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:14.099234165 +0000 UTC m=+23.743310859" watchObservedRunningTime="2026-09-08 16:13:14.991831169 +0000 UTC m=+24.635907864"
Sep 08 16:13:15.080219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.080195    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:13:15.081233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.081203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"4b28deaba136555365158725da094813e63d67964ed01698926171f673b0f5d9"}
Sep 08 16:13:15.083561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.083537    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h" event={"ID":"e3e11b0e-11fc-438b-ae4b-867e1a16e713","Type":"ContainerStarted","Data":"81cb2f5add27a0790d738604343012d2cca9f8757b7e4c007ec1a4cd8436ed28"}
Sep 08 16:13:15.084182 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.084146    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:13:15.084618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.084600    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8cnpg"
Sep 08 16:13:15.116234 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.116194    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slt4h" podStartSLOduration=4.770573024 podStartE2EDuration="25.116179329s" totalImagesPullingTime="20.345606305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:15.11566205 +0000 UTC m=+24.759738743" watchObservedRunningTime="2026-09-08 16:13:15.116179329 +0000 UTC m=+24.760256021"
Sep 08 16:13:15.882088 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.882059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:15.882256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:15.882098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:15.882256 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:15.882153    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:15.882351 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:15.882298    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:16.882351 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:16.882269    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:16.882808 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:16.882409    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:17.095624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.094930    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:13:17.095796 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.095754    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"2e764530fa81b1965de1baafde31df5cc4fa7f4b4d98e90269298bd19d01ef43"}
Sep 08 16:13:17.095865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.095851    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:17.095919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.095908    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:17.095963 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.095925    2571 scope.go:117] "RemoveContainer" containerID="3f0604315c8b20eedd00f3aef34151fde56bfe8454d3cd2e75480ed5a7524e98"
Sep 08 16:13:17.095963 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.095957    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:17.115356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.115041    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:17.119721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.119696    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:17.882748 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.882537    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:17.883202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:17.882597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:17.883202 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:17.882858    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:17.883202 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:17.882879    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:18.100850 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.100828    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:13:18.101228 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.101199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" event={"ID":"1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5","Type":"ContainerStarted","Data":"1960331fa122ea295f7ce322e7ec2defaf5bd7f4035c61c0d678390586936706"}
Sep 08 16:13:18.103087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.103063    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="6406917daafa61c055548ef42cac293cbfd777191903b7659d9b7bb4fc9f54e4" exitCode=0
Sep 08 16:13:18.103175 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.103112    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"6406917daafa61c055548ef42cac293cbfd777191903b7659d9b7bb4fc9f54e4"}
Sep 08 16:13:18.130415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.130374    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv" podStartSLOduration=9.443882086 podStartE2EDuration="28.130362551s" totalImagesPullingTime="18.686480465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:18.129882082 +0000 UTC m=+27.773958775" watchObservedRunningTime="2026-09-08 16:13:18.130362551 +0000 UTC m=+27.774439264"
Sep 08 16:13:18.883798 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.883167    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:18.883798 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:18.883305    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:18.930541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.930513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6xhh"]
Sep 08 16:13:18.930642 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.930630    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:18.930723 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:18.930706    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:18.933382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.933361    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkt2m"]
Sep 08 16:13:18.933475 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.933445    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:18.933573 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:18.933553    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:18.934319 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:18.934299    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vhdw"]
Sep 08 16:13:19.107074 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:19.107044    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="9fe6c5af3032e82408cac84bfbfaba6e14f124f5e8cf20f6041c01b85c1fc039" exitCode=0
Sep 08 16:13:19.107211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:19.107120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"9fe6c5af3032e82408cac84bfbfaba6e14f124f5e8cf20f6041c01b85c1fc039"}
Sep 08 16:13:19.107211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:19.107160    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:19.108042 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:19.107644    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:20.882966 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:20.882804    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:20.883361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:20.882869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:20.883361 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:20.883133    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:20.883361 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:20.883030    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:20.883361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:20.882899    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:20.883361 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:20.883243    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:21.114040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:21.114008    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="42b550f87024ff7776aa1d509f4da10b8ec6a7d9db10edfdbccb45e184eb3c3c" exitCode=0
Sep 08 16:13:21.114183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:21.114093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"42b550f87024ff7776aa1d509f4da10b8ec6a7d9db10edfdbccb45e184eb3c3c"}
Sep 08 16:13:22.882489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:22.882406    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:22.882489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:22.882442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:22.882922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:22.882419    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:22.882922 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:22.882546    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:22.882922 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:22.882672    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:22.882922 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:22.882749    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:24.493710 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.493673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:24.494226 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.493856    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:24.494226 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.493932    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret podName:0e0303ba-eecf-48b6-887a-87fcf3f78535 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:56.49391126 +0000 UTC m=+66.137987945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret") pod "global-pull-secret-syncer-f6xhh" (UID: "0e0303ba-eecf-48b6-887a-87fcf3f78535") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:24.594652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.594621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:24.594828 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.594812    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:24.594888 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.594879    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:13:56.594859592 +0000 UTC m=+66.238936268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:24.695278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.695239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:24.695461 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.695443    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:13:24.695529 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.695468    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:13:24.695529 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.695482    2571 projected.go:196] Error preparing data for projected volume kube-api-access-jr7rd for pod openshift-network-diagnostics/network-check-target-gkt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:24.695632 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.695549    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd podName:dd04b6da-62ec-4c0e-a940-929cd5da2e57 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:56.695528527 +0000 UTC m=+66.339605214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7rd" (UniqueName: "kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd") pod "network-check-target-gkt2m" (UID: "dd04b6da-62ec-4c0e-a940-929cd5da2e57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:24.881887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.881852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:24.882075 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.881995    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:13:24.882390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.882363    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:24.882519 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.882468    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkt2m" podUID="dd04b6da-62ec-4c0e-a940-929cd5da2e57"
Sep 08 16:13:24.882519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:24.882480    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:24.882625 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:24.882571    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6xhh" podUID="0e0303ba-eecf-48b6-887a-87fcf3f78535"
Sep 08 16:13:25.071068 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.071041    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-163.ec2.internal" event="NodeReady"
Sep 08 16:13:25.071248 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.071177    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 16:13:25.111917 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.111883    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cdhsw"]
Sep 08 16:13:25.147389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.147312    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xqgfn"]
Sep 08 16:13:25.147541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.147451    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.150371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.150347    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 16:13:25.150503 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.150347    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 16:13:25.150570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.150539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\""
Sep 08 16:13:25.166742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.166707    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cdhsw"]
Sep 08 16:13:25.166862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.166749    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqgfn"]
Sep 08 16:13:25.166862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.166846    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.169555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.169337    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 16:13:25.169555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.169392    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 16:13:25.169555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.169449    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7rjg\""
Sep 08 16:13:25.169555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.169397    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 16:13:25.301285 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbtf\" (UniqueName: \"kubernetes.io/projected/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-kube-api-access-lwbtf\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.301467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301311    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.301467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-cert\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.301467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmbx\" (UniqueName: \"kubernetes.io/projected/e6a6631c-8c75-47b9-aec8-2bafa218cc67-kube-api-access-vlmbx\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.301614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6631c-8c75-47b9-aec8-2bafa218cc67-config-volume\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.301614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.301506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6631c-8c75-47b9-aec8-2bafa218cc67-tmp-dir\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.402525 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmbx\" (UniqueName: \"kubernetes.io/projected/e6a6631c-8c75-47b9-aec8-2bafa218cc67-kube-api-access-vlmbx\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.402525 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6631c-8c75-47b9-aec8-2bafa218cc67-config-volume\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.402795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6631c-8c75-47b9-aec8-2bafa218cc67-tmp-dir\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.402795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbtf\" (UniqueName: \"kubernetes.io/projected/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-kube-api-access-lwbtf\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.402795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.402795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-cert\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.402988 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:25.402797    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:25.402988 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:25.402868    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:25.902846111 +0000 UTC m=+35.546922781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:25.402988 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.402936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a6631c-8c75-47b9-aec8-2bafa218cc67-tmp-dir\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.403150 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.403128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6631c-8c75-47b9-aec8-2bafa218cc67-config-volume\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.407421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.407367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-cert\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.411348 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.411323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmbx\" (UniqueName: \"kubernetes.io/projected/e6a6631c-8c75-47b9-aec8-2bafa218cc67-kube-api-access-vlmbx\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.416352 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.416317    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:25.418476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.418455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbtf\" (UniqueName: \"kubernetes.io/projected/ded0bbca-b5d7-4dce-80ad-44ce042b3c48-kube-api-access-lwbtf\") pod \"ingress-canary-xqgfn\" (UID: \"ded0bbca-b5d7-4dce-80ad-44ce042b3c48\") " pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.475605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.475570    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqgfn"
Sep 08 16:13:25.496492 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.496465    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:25.864116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.863916    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqgfn"]
Sep 08 16:13:25.868432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.868402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:25.876599 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:13:25.876560    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded0bbca_b5d7_4dce_80ad_44ce042b3c48.slice/crio-de89ad000257cccad39355be4f749bf8a53fde852d4429794e9f822ee635ceff WatchSource:0}: Error finding container de89ad000257cccad39355be4f749bf8a53fde852d4429794e9f822ee635ceff: Status 404 returned error can't find the container with id de89ad000257cccad39355be4f749bf8a53fde852d4429794e9f822ee635ceff
Sep 08 16:13:25.906556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.906530    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:25.906708 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:25.906689    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:25.906761 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:25.906752    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:26.906736953 +0000 UTC m=+36.550813622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:25.935577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.935510    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:25.995014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:25.994979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:26.048363 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.048331    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:26.129175 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.129145    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="5382770b7e645bedb75c01858d7f4fce562cdd14bcf8cd63d0c695ca26bb6222" exitCode=0
Sep 08 16:13:26.129374 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.129207    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"5382770b7e645bedb75c01858d7f4fce562cdd14bcf8cd63d0c695ca26bb6222"}
Sep 08 16:13:26.130454 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.130435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqgfn" event={"ID":"ded0bbca-b5d7-4dce-80ad-44ce042b3c48","Type":"ContainerStarted","Data":"de89ad000257cccad39355be4f749bf8a53fde852d4429794e9f822ee635ceff"}
Sep 08 16:13:26.280634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.280594    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:26.337222 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.337185    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:26.885674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.885645    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:26.886559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.885744    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:26.886559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.885784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:26.889357 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.889331    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:13:26.889357 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.889339    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:13:26.889542 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.889339    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:13:26.889542 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.889341    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\""
Sep 08 16:13:26.890552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.890423    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\""
Sep 08 16:13:26.890552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.890448    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:13:26.915349 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:26.915327    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:26.915511 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:26.915455    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:26.915660 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:26.915519    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:28.915504576 +0000 UTC m=+38.559581245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:27.137440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:27.137344    2571 generic.go:356] "Generic (PLEG): container finished" podID="434f72bb-ef9a-4917-8bf1-bf0ff3e9258b" containerID="b4a0fe4ab2e054c47dd1610eef3a64f986079027bf304c46ff486f645859b5f0" exitCode=0
Sep 08 16:13:27.137440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:27.137413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerDied","Data":"b4a0fe4ab2e054c47dd1610eef3a64f986079027bf304c46ff486f645859b5f0"}
Sep 08 16:13:28.144262 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:28.144220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" event={"ID":"434f72bb-ef9a-4917-8bf1-bf0ff3e9258b","Type":"ContainerStarted","Data":"4a7ddffd35a25dc4fd00acddde11a592335e760e54d03ab8870950d27194a32a"}
Sep 08 16:13:28.179510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:28.179458    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kbt7b" podStartSLOduration=8.601966987 podStartE2EDuration="38.179444392s" totalImagesPullingTime="29.577477405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:28.178005411 +0000 UTC m=+37.822082113" watchObservedRunningTime="2026-09-08 16:13:28.179444392 +0000 UTC m=+37.823521122"
Sep 08 16:13:28.298348 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:28.298312    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:28.348369 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:28.348332    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:28.930479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:28.930282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:28.930479 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:28.930472    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:28.930729 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:28.930564    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:32.930542705 +0000 UTC m=+42.574619393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:29.148466 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:29.148429    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqgfn" event={"ID":"ded0bbca-b5d7-4dce-80ad-44ce042b3c48","Type":"ContainerStarted","Data":"184e10c383413604dbb97e1dae3cb90c90fe848cebf17ebd2fbd931b74f75fae"}
Sep 08 16:13:29.163381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:29.163338    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xqgfn" podStartSLOduration=1.556432332 podStartE2EDuration="4.163322256s" totalImagesPullingTime="2.606889924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:29.163137672 +0000 UTC m=+38.807214363" watchObservedRunningTime="2026-09-08 16:13:29.163322256 +0000 UTC m=+38.807398948"
Sep 08 16:13:30.243199 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:30.243165    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:30.299532 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:30.299489    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:30.397103 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:30.397068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:30.476533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:30.476493    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:32.960412 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:32.960367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:32.960831 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:32.960485    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:32.960831 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:32.960539    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:40.960525785 +0000 UTC m=+50.604602455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:41.013039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:41.012994    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:41.013450 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:41.013130    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:41.013450 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:41.013193    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:57.013179532 +0000 UTC m=+66.657256202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:45.858587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:45.858547    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:45.959192 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:45.959154    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:47.061749 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.061718    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"]
Sep 08 16:13:47.063548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.063533    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.066166 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.066141    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 16:13:47.066288 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.066176    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 16:13:47.067162 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.067141    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 16:13:47.067296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.067159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 16:13:47.067296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.067188    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 16:13:47.067447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.067384    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 16:13:47.067507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.067492    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 16:13:47.074534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.074513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"]
Sep 08 16:13:47.159458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.159458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.159712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.159712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.159712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.159712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.159633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm48q\" (UniqueName: \"kubernetes.io/projected/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-kube-api-access-wm48q\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.254160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.254131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:47.260063 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.260218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260151    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.260218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm48q\" (UniqueName: \"kubernetes.io/projected/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-kube-api-access-wm48q\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.260434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.261003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.260954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.263656 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.263623    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-ca\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.263746 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.263679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.263811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.263748    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.263857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.263836    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-hub\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.268185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.268158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm48q\" (UniqueName: \"kubernetes.io/projected/7d4a0352-ac9b-4f54-aed6-b8d56567c0a3-kube-api-access-wm48q\") pod \"cluster-proxy-proxy-agent-5985dcd66d-dh25g\" (UID: \"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.306241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.306198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:47.382862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.382826    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:13:47.655237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:47.655169    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"]
Sep 08 16:13:47.658366 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:13:47.658340    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d4a0352_ac9b_4f54_aed6_b8d56567c0a3.slice/crio-8af6f864ba39ec1929f4a19f441ff1b59f05387d6ddcef0d9ba7f4b9233d5a19 WatchSource:0}: Error finding container 8af6f864ba39ec1929f4a19f441ff1b59f05387d6ddcef0d9ba7f4b9233d5a19: Status 404 returned error can't find the container with id 8af6f864ba39ec1929f4a19f441ff1b59f05387d6ddcef0d9ba7f4b9233d5a19
Sep 08 16:13:48.195284 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:48.195251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerStarted","Data":"8af6f864ba39ec1929f4a19f441ff1b59f05387d6ddcef0d9ba7f4b9233d5a19"}
Sep 08 16:13:49.129023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:49.128994    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp7kv"
Sep 08 16:13:49.244762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:49.244724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:49.302159 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:49.302115    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:13:53.212640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:53.212586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerStarted","Data":"24b2ce57f8d86584993d833128b75e42f779e4a1dfd6c436847de57c19875bc6"}
Sep 08 16:13:53.212640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:53.212643    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerStarted","Data":"573c71101fb45ec7db2fc666ff0bbed89fcfb9c5dbc1ecdc95c69c43a82e0f0e"}
Sep 08 16:13:53.213231 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:53.212661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerStarted","Data":"300dd2dfeeca07ef79cae3c21f1e097a74daa92b042f1daf282848cf08042418"}
Sep 08 16:13:53.232106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:53.232050    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" podStartSLOduration=1.562944476 podStartE2EDuration="6.232035993s" totalImagesPullingTime="4.669091517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:53.230560406 +0000 UTC m=+62.874637097" watchObservedRunningTime="2026-09-08 16:13:53.232035993 +0000 UTC m=+62.876112684"
Sep 08 16:13:56.524249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.524214    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:56.527179 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.527159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:13:56.537365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.537345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e0303ba-eecf-48b6-887a-87fcf3f78535-original-pull-secret\") pod \"global-pull-secret-syncer-f6xhh\" (UID: \"0e0303ba-eecf-48b6-887a-87fcf3f78535\") " pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:56.597506 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.597463    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6xhh"
Sep 08 16:13:56.625168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.625133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:13:56.627881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.627860    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:13:56.635844 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:56.635825    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 16:13:56.635983 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:56.635894    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:15:00.635871691 +0000 UTC m=+130.279948364 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : secret "metrics-daemon-secret" not found
Sep 08 16:13:56.726256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.726221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:56.729223 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.729201    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:13:56.739189 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.739169    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:13:56.750633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.750595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7rd\" (UniqueName: \"kubernetes.io/projected/dd04b6da-62ec-4c0e-a940-929cd5da2e57-kube-api-access-jr7rd\") pod \"network-check-target-gkt2m\" (UID: \"dd04b6da-62ec-4c0e-a940-929cd5da2e57\") " pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:56.893507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.893464    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6xhh"]
Sep 08 16:13:56.898198 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:13:56.898168    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0303ba_eecf_48b6_887a_87fcf3f78535.slice/crio-e6ce9a505379baf0539bbfb47483d64906301785f8fd7fcd8612ed928819b93e WatchSource:0}: Error finding container e6ce9a505379baf0539bbfb47483d64906301785f8fd7fcd8612ed928819b93e: Status 404 returned error can't find the container with id e6ce9a505379baf0539bbfb47483d64906301785f8fd7fcd8612ed928819b93e
Sep 08 16:13:56.912156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.912130    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\""
Sep 08 16:13:56.920229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:56.920210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:13:57.028446 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:57.028418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:13:57.028579 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:57.028531    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:57.028621 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:13:57.028582    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:14:29.028569566 +0000 UTC m=+98.672646235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:57.186170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:57.186098    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkt2m"]
Sep 08 16:13:57.188958 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:13:57.188921    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd04b6da_62ec_4c0e_a940_929cd5da2e57.slice/crio-6ee39b79c3237ab29a2e23eb254317e04438f07c1857492a01478a5b937540a0 WatchSource:0}: Error finding container 6ee39b79c3237ab29a2e23eb254317e04438f07c1857492a01478a5b937540a0: Status 404 returned error can't find the container with id 6ee39b79c3237ab29a2e23eb254317e04438f07c1857492a01478a5b937540a0
Sep 08 16:13:57.223368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:57.223339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkt2m" event={"ID":"dd04b6da-62ec-4c0e-a940-929cd5da2e57","Type":"ContainerStarted","Data":"6ee39b79c3237ab29a2e23eb254317e04438f07c1857492a01478a5b937540a0"}
Sep 08 16:13:57.224442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:13:57.224421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6xhh" event={"ID":"0e0303ba-eecf-48b6-887a-87fcf3f78535","Type":"ContainerStarted","Data":"e6ce9a505379baf0539bbfb47483d64906301785f8fd7fcd8612ed928819b93e"}
Sep 08 16:14:01.440528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:01.440490    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:14:01.498029 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:01.498000    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:14:02.239788 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:02.239730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6xhh" event={"ID":"0e0303ba-eecf-48b6-887a-87fcf3f78535","Type":"ContainerStarted","Data":"7fde63c17d4983b8d4cff6008d198960a93530cc1e1902d1faec2eb85565371b"}
Sep 08 16:14:02.241227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:02.241201    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkt2m" event={"ID":"dd04b6da-62ec-4c0e-a940-929cd5da2e57","Type":"ContainerStarted","Data":"bdcbec96a39e687f5a6d75dee6650bcbc19c4328f700be956d54e07450330255"}
Sep 08 16:14:02.241338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:02.241328    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:14:02.257024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:02.256984    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f6xhh" podStartSLOduration=67.66060615 podStartE2EDuration="1m12.256969045s" totalImagesPullingTime="4.596362895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:14:02.256454141 +0000 UTC m=+71.900530832" watchObservedRunningTime="2026-09-08 16:14:02.256969045 +0000 UTC m=+71.901045736"
Sep 08 16:14:02.269953 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:02.269904    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gkt2m" podStartSLOduration=67.967544811 podStartE2EDuration="1m12.269889048s" totalImagesPullingTime="4.302344237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:14:02.2693281 +0000 UTC m=+71.913404791" watchObservedRunningTime="2026-09-08 16:14:02.269889048 +0000 UTC m=+71.913965740"
Sep 08 16:14:29.041405 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:29.041357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:14:29.041956 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:14:29.041484    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:14:29.041956 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:14:29.041572    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:15:33.041550543 +0000 UTC m=+162.685627214 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:14:33.248609 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:14:33.248576    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gkt2m"
Sep 08 16:15:00.656152 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:15:00.656114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:15:00.656722 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:00.656319    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 16:15:00.656722 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:00.656423    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs podName:ecd65ad8-a513-44bf-9bdc-986b9e57190e nodeName:}" failed. No retries permitted until 2026-09-08 16:17:02.656400018 +0000 UTC m=+252.300476693 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs") pod "network-metrics-daemon-7vhdw" (UID: "ecd65ad8-a513-44bf-9bdc-986b9e57190e") : secret "metrics-daemon-secret" not found
Sep 08 16:15:28.158097 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:28.158041    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cdhsw" podUID="e6a6631c-8c75-47b9-aec8-2bafa218cc67"
Sep 08 16:15:28.491634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:15:28.491608    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:15:29.903502 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:29.903458    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7vhdw" podUID="ecd65ad8-a513-44bf-9bdc-986b9e57190e"
Sep 08 16:15:33.072985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:15:33.072947    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:15:33.073372 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:33.073053    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:15:33.073372 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:15:33.073114    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls podName:e6a6631c-8c75-47b9-aec8-2bafa218cc67 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:35.073100467 +0000 UTC m=+284.717177137 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls") pod "dns-default-cdhsw" (UID: "e6a6631c-8c75-47b9-aec8-2bafa218cc67") : secret "dns-default-metrics-tls" not found
Sep 08 16:15:43.882017 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:15:43.881923    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:16:08.008683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:08.008649    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:35.065543 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:35.065504    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:35.167989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:35.167960    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:38.828339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:38.828304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:38.917974 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:38.917947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:39.563313 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.563281    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:39.664488 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.664459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:39.711280 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.711254    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pthx9"]
Sep 08 16:16:39.713228 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.713213    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.716652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.716627    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 16:16:39.717242 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.717228    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 16:16:39.717714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.717698    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\""
Sep 08 16:16:39.717872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.717854    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 16:16:39.717939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.717907    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 16:16:39.730879 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.730858    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pthx9"]
Sep 08 16:16:39.767547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.767522    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:39.812534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.812511    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9267b4c3-cb2b-4882-8c54-e0d692155618-crio-socket\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.812628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.812535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clt8w\" (UniqueName: \"kubernetes.io/projected/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-api-access-clt8w\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.812628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.812568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.812709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.812641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9267b4c3-cb2b-4882-8c54-e0d692155618-data-volume\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.812709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.812666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9267b4c3-cb2b-4882-8c54-e0d692155618-crio-socket\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clt8w\" (UniqueName: \"kubernetes.io/projected/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-api-access-clt8w\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913260    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9267b4c3-cb2b-4882-8c54-e0d692155618-data-volume\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9267b4c3-cb2b-4882-8c54-e0d692155618-crio-socket\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:39.913324    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:39.913388    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.413369964 +0000 UTC m=+230.057446634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:39.913461    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:39.913513 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:39.913513    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.413496421 +0000 UTC m=+230.057573095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:39.913835 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.913564    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9267b4c3-cb2b-4882-8c54-e0d692155618-data-volume\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:39.923310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:39.923285    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clt8w\" (UniqueName: \"kubernetes.io/projected/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-api-access-clt8w\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:40.111987 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.111951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:40.197333 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.197254    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:40.418078 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.418029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:40.418078 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.418093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:40.418295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:40.418191    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:40.418295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:40.418265    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.418248612 +0000 UTC m=+231.062325282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:40.418295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:40.418195    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:40.418404 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:40.418333    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.41832127 +0000 UTC m=+231.062397939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:40.592340 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.592298    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:40.689296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.689261    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:40.890588 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:40.890502    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:41.271624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:41.271523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:41.424910 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:41.424878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:41.425116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:41.424946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:41.425116 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:41.425006    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:41.425116 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:41.425066    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.425051494 +0000 UTC m=+233.069128183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:41.425116 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:41.425073    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:41.425290 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:41.425133    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.425118415 +0000 UTC m=+233.069195100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:42.726489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:42.726452    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:42.764488 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:42.764468    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vw7sb_0e3586bf-382f-44e6-90bd-7dd42f1489f8/dns-node-resolver/0.log"
Sep 08 16:16:43.435738 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:43.435704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:43.435973 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:43.435789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:43.435973 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:43.435882    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:43.435973 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:43.435883    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:43.435973 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:43.435958    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.435939343 +0000 UTC m=+237.080016014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:43.435973 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:43.435973    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.435966311 +0000 UTC m=+237.080042980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:43.765601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:43.765529    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ss6b4_b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574/node-ca/0.log"
Sep 08 16:16:44.289496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:44.289458    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:44.565106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:44.565041    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqgfn_ded0bbca-b5d7-4dce-80ad-44ce042b3c48/serve-healthcheck-canary/0.log"
Sep 08 16:16:46.334802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:46.334752    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:46.402297 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:46.402262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:46.895731 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:46.895697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:47.002501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:47.002468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:47.179918 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:47.179829    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:47.463319 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:47.463211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:47.463319 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:47.463283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:47.463736 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:47.463340    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:47.463736 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:47.463391    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:47.463736 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:47.463409    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.463392672 +0000 UTC m=+245.107469342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:47.463736 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:47.463444    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.463428306 +0000 UTC m=+245.107504978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:49.903293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:49.903258    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.003999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.003964    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.223071 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.222968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.322603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.322570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.580251 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.580214    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.680443 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.680406    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.784698 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.784663    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:50.994465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:50.994390    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:51.185274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:51.185240    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:52.178974 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:52.178943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:52.380337 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:52.380308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:52.721282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:52.721243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:52.824200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:52.824167    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:52.922431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:52.922395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:53.022571 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:53.022484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:53.984982 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:53.984944    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:54.182180 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:54.182144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:54.380651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:54.380621    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:54.580212 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:54.580177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:54.783802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:54.783702    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:55.521257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:55.521223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:55.521631 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:55.521279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:55.521631 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:55.521408    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:55.521631 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:16:55.521491    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls podName:9267b4c3-cb2b-4882-8c54-e0d692155618 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:11.521471241 +0000 UTC m=+261.165547912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pthx9" (UID: "9267b4c3-cb2b-4882-8c54-e0d692155618") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:55.521906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:55.521887    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9267b4c3-cb2b-4882-8c54-e0d692155618-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:16:55.778301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:55.778216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:56.249388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:56.249359    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:56.390132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:56.390093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:56.589388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:56.589357    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:56.988238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:56.988118    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:16:57.183230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:16:57.183197    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:00.502357 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:00.502322    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:00.567697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:00.567667    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:02.672992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:02.672936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:17:02.675440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:02.675407    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd65ad8-a513-44bf-9bdc-986b9e57190e-metrics-certs\") pod \"network-metrics-daemon-7vhdw\" (UID: \"ecd65ad8-a513-44bf-9bdc-986b9e57190e\") " pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:17:02.786300 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:02.786263    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\""
Sep 08 16:17:02.793490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:02.793469    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhdw"
Sep 08 16:17:03.064211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:03.064131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vhdw"]
Sep 08 16:17:03.067165 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:03.067126    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd65ad8_a513_44bf_9bdc_986b9e57190e.slice/crio-8067ef3c1c1a028e51bde7148075489cae4a97579ff2232724b273391adfc42f WatchSource:0}: Error finding container 8067ef3c1c1a028e51bde7148075489cae4a97579ff2232724b273391adfc42f: Status 404 returned error can't find the container with id 8067ef3c1c1a028e51bde7148075489cae4a97579ff2232724b273391adfc42f
Sep 08 16:17:03.765372 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:03.765331    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhdw" event={"ID":"ecd65ad8-a513-44bf-9bdc-986b9e57190e","Type":"ContainerStarted","Data":"8067ef3c1c1a028e51bde7148075489cae4a97579ff2232724b273391adfc42f"}
Sep 08 16:17:04.769760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:04.769713    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhdw" event={"ID":"ecd65ad8-a513-44bf-9bdc-986b9e57190e","Type":"ContainerStarted","Data":"8fdfe57c787db8c6a533ef30fd373ada4dd6f04ddf4582dc749605fc8fcaa281"}
Sep 08 16:17:04.770156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:04.769799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhdw" event={"ID":"ecd65ad8-a513-44bf-9bdc-986b9e57190e","Type":"ContainerStarted","Data":"f698bdd51840697f80981b0cebbda8b9775669d6569f4b0b7ef861fdc0795a02"}
Sep 08 16:17:04.785637 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:04.785581    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vhdw" podStartSLOduration=253.931693601 podStartE2EDuration="4m14.785563493s" totalImagesPullingTime="853.869892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:04.784965777 +0000 UTC m=+254.429042469" watchObservedRunningTime="2026-09-08 16:17:04.785563493 +0000 UTC m=+254.429640187"
Sep 08 16:17:05.542437 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.542376    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5789b9fbd6-4kdw6"]
Sep 08 16:17:05.544455 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.544433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.548332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.548301    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 16:17:05.548460 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.548386    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgmfx\""
Sep 08 16:17:05.548880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.548867    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 16:17:05.549707 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.549681    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 16:17:05.554081 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.554061    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 16:17:05.566011 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.565985    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5789b9fbd6-4kdw6"]
Sep 08 16:17:05.581507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.581470    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:05.593287 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-image-registry-private-configuration\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-trusted-ca\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593329    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-bound-sa-token\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-registry-tls\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9065f588-b363-4690-a1ed-899fbc692584-ca-trust-extracted\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593479    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbkk\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-kube-api-access-qxbkk\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-registry-certificates\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.593644 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.593573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-installation-pull-secrets\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9065f588-b363-4690-a1ed-899fbc692584-ca-trust-extracted\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbkk\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-kube-api-access-qxbkk\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-registry-certificates\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-installation-pull-secrets\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-image-registry-private-configuration\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-trusted-ca\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.694756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-bound-sa-token\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.695080 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.694819    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-registry-tls\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.695080 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.695040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9065f588-b363-4690-a1ed-899fbc692584-ca-trust-extracted\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.695531 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.695465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-registry-certificates\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.696415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.696387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9065f588-b363-4690-a1ed-899fbc692584-trusted-ca\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.697517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.697499    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-image-registry-private-configuration\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.697589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.697538    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9065f588-b363-4690-a1ed-899fbc692584-installation-pull-secrets\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.697589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.697578    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-registry-tls\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.704101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.704077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-bound-sa-token\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.704226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.704210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbkk\" (UniqueName: \"kubernetes.io/projected/9065f588-b363-4690-a1ed-899fbc692584-kube-api-access-qxbkk\") pod \"image-registry-5789b9fbd6-4kdw6\" (UID: \"9065f588-b363-4690-a1ed-899fbc692584\") " pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:05.706184 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.706158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:05.806058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.805959    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:05.853242 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:05.853204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:06.129382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:06.129352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5789b9fbd6-4kdw6"]
Sep 08 16:17:06.133063 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:06.133036    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9065f588_b363_4690_a1ed_899fbc692584.slice/crio-d9777ba02235ae2a5f2eaf48ea9149b4672936b158050e146a7577e97a2a6ed7 WatchSource:0}: Error finding container d9777ba02235ae2a5f2eaf48ea9149b4672936b158050e146a7577e97a2a6ed7: Status 404 returned error can't find the container with id d9777ba02235ae2a5f2eaf48ea9149b4672936b158050e146a7577e97a2a6ed7
Sep 08 16:17:06.777325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:06.777283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" event={"ID":"9065f588-b363-4690-a1ed-899fbc692584","Type":"ContainerStarted","Data":"e4fb8061dcb11c531056f010bca7f257db40492ea4bac604e2492fff882f9006"}
Sep 08 16:17:06.777477 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:06.777340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" event={"ID":"9065f588-b363-4690-a1ed-899fbc692584","Type":"ContainerStarted","Data":"d9777ba02235ae2a5f2eaf48ea9149b4672936b158050e146a7577e97a2a6ed7"}
Sep 08 16:17:06.777477 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:06.777410    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:06.798675 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:06.798627    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podStartSLOduration=1.798612901 podStartE2EDuration="1.798612901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:06.797338884 +0000 UTC m=+256.441415575" watchObservedRunningTime="2026-09-08 16:17:06.798612901 +0000 UTC m=+256.442689590"
Sep 08 16:17:09.238174 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.238120    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:09.339595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.339563    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:09.417504 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.417466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:09.499383 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.499302    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:09.892480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.892450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:09.955751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:09.955723    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:11.247445 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:11.247412    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:11.538023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:11.537908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:17:11.540287 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:11.540268    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9267b4c3-cb2b-4882-8c54-e0d692155618-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pthx9\" (UID: \"9267b4c3-cb2b-4882-8c54-e0d692155618\") " pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:17:11.827017 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:11.826990    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\""
Sep 08 16:17:11.835805 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:11.835787    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pthx9"
Sep 08 16:17:12.106136 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.106036    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pthx9"]
Sep 08 16:17:12.110653 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:12.110631    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9267b4c3_cb2b_4882_8c54_e0d692155618.slice/crio-5f4136e4841d3547bbd80edac9d601d2b65e02b1e7539e41e2cf66b812903567 WatchSource:0}: Error finding container 5f4136e4841d3547bbd80edac9d601d2b65e02b1e7539e41e2cf66b812903567: Status 404 returned error can't find the container with id 5f4136e4841d3547bbd80edac9d601d2b65e02b1e7539e41e2cf66b812903567
Sep 08 16:17:12.411610 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.411525    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:12.473873 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.473847    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:12.525367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.525335    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:12.799087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.799012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pthx9" event={"ID":"9267b4c3-cb2b-4882-8c54-e0d692155618","Type":"ContainerStarted","Data":"ee3f8cac1cd3658770ef2ac05d96c88659714afc53bc4f8a2fc429a2bfa1e5a1"}
Sep 08 16:17:12.799087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:12.799068    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pthx9" event={"ID":"9267b4c3-cb2b-4882-8c54-e0d692155618","Type":"ContainerStarted","Data":"5f4136e4841d3547bbd80edac9d601d2b65e02b1e7539e41e2cf66b812903567"}
Sep 08 16:17:14.806231 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:14.806194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pthx9" event={"ID":"9267b4c3-cb2b-4882-8c54-e0d692155618","Type":"ContainerStarted","Data":"a0f1a2937389f0ca40c0a44c656910fc9c98475c69f53724a2b146153d98b446"}
Sep 08 16:17:15.029218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:15.029179    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:16.786947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.786917    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:16.820285 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.820245    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"]
Sep 08 16:17:16.823594 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.823570    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.825900 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.825873    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 16:17:16.826004 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.825920    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kh4ss\""
Sep 08 16:17:16.827149 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.827128    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 16:17:16.827259 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.827151    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 16:17:16.827259 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.827161    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 16:17:16.827259 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.827220    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 16:17:16.827259 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.827128    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 16:17:16.837787 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.837748    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-86v6p"]
Sep 08 16:17:16.840783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.840753    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.843193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.843076    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 16:17:16.843193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.843158    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"]
Sep 08 16:17:16.843361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.843235    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 16:17:16.843361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.843290    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 16:17:16.843361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.843320    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dqlk\""
Sep 08 16:17:16.884979 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.884946    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:16.974840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-wtmp\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.974840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-metrics-client-ca\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974909    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-textfile\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-sys\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.974961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-root\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975045    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhg7\" (UniqueName: \"kubernetes.io/projected/711a050c-ab0c-4535-93d4-63153e8544dc-kube-api-access-pjhg7\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/711a050c-ab0c-4535-93d4-63153e8544dc-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-tls\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975134    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:16.975311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:16.975174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5k6r\" (UniqueName: \"kubernetes.io/projected/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-kube-api-access-l5k6r\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.075910 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-textfile\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075937    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-sys\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-root\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.075990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhg7\" (UniqueName: \"kubernetes.io/projected/711a050c-ab0c-4535-93d4-63153e8544dc-kube-api-access-pjhg7\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076030    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-sys\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/711a050c-ab0c-4535-93d4-63153e8544dc-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-root\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-tls\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076154    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6r\" (UniqueName: \"kubernetes.io/projected/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-kube-api-access-l5k6r\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:17:17.076172    2571 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-wtmp\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:17:17.076238    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls podName:711a050c-ab0c-4535-93d4-63153e8544dc nodeName:}" failed. No retries permitted until 2026-09-08 16:17:17.576218256 +0000 UTC m=+267.220294941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-2xq8w" (UID: "711a050c-ab0c-4535-93d4-63153e8544dc") : secret "kube-state-metrics-tls" not found
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-textfile\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076287    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-metrics-client-ca\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076335    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/711a050c-ab0c-4535-93d4-63153e8544dc-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.076593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.077066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076654    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-wtmp\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.077066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.076914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-metrics-client-ca\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.077136 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.077086    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.077136 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.077119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711a050c-ab0c-4535-93d4-63153e8544dc-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.078513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.078490    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.078685 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.078664    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.079052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.079037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-node-exporter-tls\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.086564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.086540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhg7\" (UniqueName: \"kubernetes.io/projected/711a050c-ab0c-4535-93d4-63153e8544dc-kube-api-access-pjhg7\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.086939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.086919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5k6r\" (UniqueName: \"kubernetes.io/projected/b8b17c31-b4fe-42a9-9d48-c62b8806e16a-kube-api-access-l5k6r\") pod \"node-exporter-86v6p\" (UID: \"b8b17c31-b4fe-42a9-9d48-c62b8806e16a\") " pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.087306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.087284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:17.151339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.151315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86v6p"
Sep 08 16:17:17.162265 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:17.162230    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b17c31_b4fe_42a9_9d48_c62b8806e16a.slice/crio-4d4ff5b98483e4b6fdea1de3a39c27e2d53a05d3e3617ab58855e55988e87fb2 WatchSource:0}: Error finding container 4d4ff5b98483e4b6fdea1de3a39c27e2d53a05d3e3617ab58855e55988e87fb2: Status 404 returned error can't find the container with id 4d4ff5b98483e4b6fdea1de3a39c27e2d53a05d3e3617ab58855e55988e87fb2
Sep 08 16:17:17.384734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.384689    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" podUID="7d4a0352-ac9b-4f54-aed6-b8d56567c0a3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 16:17:17.580939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.580906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.583506 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.583482    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711a050c-ab0c-4535-93d4-63153e8544dc-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-2xq8w\" (UID: \"711a050c-ab0c-4535-93d4-63153e8544dc\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.734726 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.734653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"
Sep 08 16:17:17.818879 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.818843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pthx9" event={"ID":"9267b4c3-cb2b-4882-8c54-e0d692155618","Type":"ContainerStarted","Data":"ecdf51b614d42d7a6be8b280e41ecc9c3c6b58ea62e4a8b6eb3c04218b4ff103"}
Sep 08 16:17:17.820212 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.820186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86v6p" event={"ID":"b8b17c31-b4fe-42a9-9d48-c62b8806e16a","Type":"ContainerStarted","Data":"4d4ff5b98483e4b6fdea1de3a39c27e2d53a05d3e3617ab58855e55988e87fb2"}
Sep 08 16:17:17.840269 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.840227    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pthx9" podStartSLOduration=34.065478991 podStartE2EDuration="38.840205609s" totalImagesPullingTime="4.774726618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:17.838935836 +0000 UTC m=+267.483012521" watchObservedRunningTime="2026-09-08 16:17:17.840205609 +0000 UTC m=+267.484282305"
Sep 08 16:17:17.868527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.868501    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:17:17.871738 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.871719    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.874094 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874069    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 16:17:17.874581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874208    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 16:17:17.874581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874073    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 16:17:17.874581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874389    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 16:17:17.874581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874392    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 16:17:17.874581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874454    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 16:17:17.874913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874729    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-prntc\""
Sep 08 16:17:17.874913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.874753    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 16:17:17.875338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.875320    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 16:17:17.875660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.875639    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 16:17:17.894481 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.894460    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:17:17.984282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984389    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crt6q\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:17.984839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:17.984820    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.016930 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.016911    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w"]
Sep 08 16:17:18.075618 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:18.075578    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711a050c_ab0c_4535_93d4_63153e8544dc.slice/crio-1bcdec35ca7631f8a4605a1769a058f5f82a3f538a3cb1239c0d6fc239af1f74 WatchSource:0}: Error finding container 1bcdec35ca7631f8a4605a1769a058f5f82a3f538a3cb1239c0d6fc239af1f74: Status 404 returned error can't find the container with id 1bcdec35ca7631f8a4605a1769a058f5f82a3f538a3cb1239c0d6fc239af1f74
Sep 08 16:17:18.085850 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.085826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crt6q\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.085946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.085866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.085946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.085923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.085945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.085974    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086298    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.086960 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.086934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.088997 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.088967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089246 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089313 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089267    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089373 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:17:18.089362    2571 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 16:17:18.089430 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:17:18.089410    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls podName:f23cafd0-e6ad-45fb-a806-2578acbbdefb nodeName:}" failed. No retries permitted until 2026-09-08 16:17:18.589393717 +0000 UTC m=+268.233470406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb") : secret "alertmanager-main-tls" not found
Sep 08 16:17:18.089789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089709    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089797    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.089789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089820    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.090167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.089941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.090167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.090017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.091348 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.091322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.094620 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.094590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crt6q\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.248554 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.248485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:18.348941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.348910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:18.408452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.408426    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:18.590136 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.590104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.593034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.593002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.781701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.781669    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:17:18.826356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.826317    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w" event={"ID":"711a050c-ab0c-4535-93d4-63153e8544dc","Type":"ContainerStarted","Data":"1bcdec35ca7631f8a4605a1769a058f5f82a3f538a3cb1239c0d6fc239af1f74"}
Sep 08 16:17:18.828174 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.828147    2571 generic.go:356] "Generic (PLEG): container finished" podID="b8b17c31-b4fe-42a9-9d48-c62b8806e16a" containerID="e75cd640f4fb4f76b4c23a3af4c5b7e02b1761206c0b3ee61eb2ea12b5e0a01e" exitCode=0
Sep 08 16:17:18.828294 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:18.828243    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86v6p" event={"ID":"b8b17c31-b4fe-42a9-9d48-c62b8806e16a","Type":"ContainerDied","Data":"e75cd640f4fb4f76b4c23a3af4c5b7e02b1761206c0b3ee61eb2ea12b5e0a01e"}
Sep 08 16:17:19.064670 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.064634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:17:19.068163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.068135    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:19.240157 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:19.240092    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23cafd0_e6ad_45fb_a806_2578acbbdefb.slice/crio-dda21a31e49f63e9d2ed49f87f3fefe2f8ff7c8d917f6d18a4699d371a1a7b0c WatchSource:0}: Error finding container dda21a31e49f63e9d2ed49f87f3fefe2f8ff7c8d917f6d18a4699d371a1a7b0c: Status 404 returned error can't find the container with id dda21a31e49f63e9d2ed49f87f3fefe2f8ff7c8d917f6d18a4699d371a1a7b0c
Sep 08 16:17:19.790448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.790412    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"]
Sep 08 16:17:19.794389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.794367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.796821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.796797    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797006    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797030    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bgqf6nr1q7lht\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797098    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4h8jn\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797102    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 16:17:19.797232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.797086    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 16:17:19.805203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.805180    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"]
Sep 08 16:17:19.834652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.834620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w" event={"ID":"711a050c-ab0c-4535-93d4-63153e8544dc","Type":"ContainerStarted","Data":"34450e416dd3a08929b28918b3c9844e7c7b8a7f38c51f58fb45072698f6ae69"}
Sep 08 16:17:19.835049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.834675    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w" event={"ID":"711a050c-ab0c-4535-93d4-63153e8544dc","Type":"ContainerStarted","Data":"06608c8a7bcfe637b659e8d37f6b51c7f34b31e2092955837582b0ec3caa5db6"}
Sep 08 16:17:19.835049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.834702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w" event={"ID":"711a050c-ab0c-4535-93d4-63153e8544dc","Type":"ContainerStarted","Data":"e93eadba7abcf595c2d5a36540e084b155fbece1812a4c37dd926365dbe8e8db"}
Sep 08 16:17:19.836115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.836090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"dda21a31e49f63e9d2ed49f87f3fefe2f8ff7c8d917f6d18a4699d371a1a7b0c"}
Sep 08 16:17:19.838430 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.838402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86v6p" event={"ID":"b8b17c31-b4fe-42a9-9d48-c62b8806e16a","Type":"ContainerStarted","Data":"f20bd125a74d8660ed1b6eb7443f81a6858cb156f5be24cc621be45de2c39941"}
Sep 08 16:17:19.838510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.838452    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86v6p" event={"ID":"b8b17c31-b4fe-42a9-9d48-c62b8806e16a","Type":"ContainerStarted","Data":"067c87455d83305cefd09f986867f8af8c1c8040cc31e653bd561a3766eb3b22"}
Sep 08 16:17:19.870780 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.870517    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-86v6p" podStartSLOduration=2.847191012 podStartE2EDuration="3.870500884s" totalImagesPullingTime="955.377647ms" totalInitContainerRuntime="67.932225ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:19.868449992 +0000 UTC m=+269.512526709" watchObservedRunningTime="2026-09-08 16:17:19.870500884 +0000 UTC m=+269.514577578"
Sep 08 16:17:19.893815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.893745    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-2xq8w" podStartSLOduration=2.688644309 podStartE2EDuration="3.893731574s" totalImagesPullingTime="1.205087265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:19.892204886 +0000 UTC m=+269.536281579" watchObservedRunningTime="2026-09-08 16:17:19.893731574 +0000 UTC m=+269.537808266"
Sep 08 16:17:19.902480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902453    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-grpc-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-metrics-client-ca\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqx9\" (UniqueName: \"kubernetes.io/projected/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-kube-api-access-dwqx9\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:19.902957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:19.902695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.003885 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.003855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.003987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-grpc-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-metrics-client-ca\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004111    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqx9\" (UniqueName: \"kubernetes.io/projected/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-kube-api-access-dwqx9\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.004335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.004219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.005206 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.005174    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-metrics-client-ca\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.007343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.007317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.007465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.007327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.007533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.007476    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.007696 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.007677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.008060 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.008037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.008564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.008547    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-secret-grpc-tls\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.013301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.013281    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqx9\" (UniqueName: \"kubernetes.io/projected/2499dbb9-fec4-46f6-9a00-b6c7c27a6425-kube-api-access-dwqx9\") pod \"thanos-querier-85fc6fcffb-l6jxt\" (UID: \"2499dbb9-fec4-46f6-9a00-b6c7c27a6425\") " pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.106633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.106607    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:20.477638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.477581    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"]
Sep 08 16:17:20.479957 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:20.479934    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2499dbb9_fec4_46f6_9a00_b6c7c27a6425.slice/crio-62d4eef009d9713741dd2d9eae54cda8c25bc2b1bcd75eff29cb047014a33e4c WatchSource:0}: Error finding container 62d4eef009d9713741dd2d9eae54cda8c25bc2b1bcd75eff29cb047014a33e4c: Status 404 returned error can't find the container with id 62d4eef009d9713741dd2d9eae54cda8c25bc2b1bcd75eff29cb047014a33e4c
Sep 08 16:17:20.595329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.595301    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:20.692230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.692198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:20.746237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.746176    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:20.843114 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.843085    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"62d4eef009d9713741dd2d9eae54cda8c25bc2b1bcd75eff29cb047014a33e4c"}
Sep 08 16:17:20.844634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.844597    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="bb11f1083f5cba342e2d5e1a6f5ee35db814f91cb595f99663c8234c01c5da79" exitCode=0
Sep 08 16:17:20.844634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:20.844627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"bb11f1083f5cba342e2d5e1a6f5ee35db814f91cb595f99663c8234c01c5da79"}
Sep 08 16:17:21.900831 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:21.900786    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:21.959339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:21.959305    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:22.285402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.285286    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:22.466633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.466503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:22.703240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.703209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:22.857687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.857657    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"51ae741071cbd15f155f7aa0dc73818ad58d4a2156f1e8fa767c7db3baa3fdc2"}
Sep 08 16:17:22.857853 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.857797    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"4bfa868b70c19298594d4187a9e2361d59153c2e82025bc484b53aa429cf52a9"}
Sep 08 16:17:22.857913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.857857    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"cfe13ef8d3b9adc79e463b15470a174a602ddf02d60ede4468a01de20e5beaa4"}
Sep 08 16:17:22.857913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.857899    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"b2ee3caa0159dc701c4b5402603e32c939dc72128b2c5c9d20fd49162210a869"}
Sep 08 16:17:22.858001 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:22.857940    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"6847f060db9ff6a8b3c577075306f56fe2ed9f15839b12a8577bd9e5bb2bf352"}
Sep 08 16:17:23.015036 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.014974    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:17:23.018811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.018789    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.019581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.019555    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:23.021411 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.021394    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4n79w\""
Sep 08 16:17:23.022017 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.021998    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 16:17:23.022089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022004    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 16:17:23.022377 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022358    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 16:17:23.022504 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022488    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 16:17:23.022573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022550    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 16:17:23.022573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022561    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 16:17:23.022802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022787    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 16:17:23.022860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.022805    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 16:17:23.023040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.023025    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 16:17:23.023106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.023094    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 16:17:23.023146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.023098    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 16:17:23.023906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.023887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 16:17:23.024415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.024402    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 16:17:23.024458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.024402    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-731mbug06c1v\""
Sep 08 16:17:23.025942 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.025925    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 16:17:23.027942 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.027923    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 16:17:23.034613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.034584    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:17:23.128959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.128875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.128959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.128908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.128959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.128932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.128959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.128957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnf7\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129456    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.129862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.129591    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230703    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.230863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.230966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnf7\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231256 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231704 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231459    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.231955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.231701    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.232298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.232271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.232740 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.232702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234168    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.234741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.235129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.234950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.235129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.235046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.235670 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.235647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.236746 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.236702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.237360 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.237335    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.237443 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.237341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.237700 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.237676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.237700 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.237678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.241300 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.241282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnf7\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7\") pod \"prometheus-k8s-0\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.329144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.329106    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:23.616678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:23.616306    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:17:24.030962 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:24.030931    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383a834b_34f2_4dfb_97b7_a03f8ff21b5e.slice/crio-fade843eb014e1ce8e5dc46903426144f7a42595157b98d508c3ee38ba7cf645 WatchSource:0}: Error finding container fade843eb014e1ce8e5dc46903426144f7a42595157b98d508c3ee38ba7cf645: Status 404 returned error can't find the container with id fade843eb014e1ce8e5dc46903426144f7a42595157b98d508c3ee38ba7cf645
Sep 08 16:17:24.868887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.868843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"c320dc9b9bfbddeaa464d4543a8d9b58ae71445d787cedb4d106b4d74e9b8a92"}
Sep 08 16:17:24.869062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.868940    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"6181275c1fdfacb8a0c1d08c80f7c7feb89787a2bf8291f08dd5a96c8baf6c99"}
Sep 08 16:17:24.869062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.868987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"d90c68fb8c93c8cf313bb7877b1a985021086218683af67f935f10a98bdf6841"}
Sep 08 16:17:24.869062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.869034    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:24.869244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.869082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"67451e2f70b47a7ccecd74c06fc27a12380f1f34566c9c71e4bebc605328bc32"}
Sep 08 16:17:24.869244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.869117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"c1b5a9cebe553833ab120bf90295309e592def359922d5e462925572916b0e0d"}
Sep 08 16:17:24.869244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.869145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" event={"ID":"2499dbb9-fec4-46f6-9a00-b6c7c27a6425","Type":"ContainerStarted","Data":"a28484762ffc2e42cd050d4c53c8100a4ec38b7a4763cdbe90b4b960c59bed98"}
Sep 08 16:17:24.870465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.870441    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3" exitCode=0
Sep 08 16:17:24.870572 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.870531    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3"}
Sep 08 16:17:24.870639 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.870625    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"fade843eb014e1ce8e5dc46903426144f7a42595157b98d508c3ee38ba7cf645"}
Sep 08 16:17:24.874193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.874173    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerStarted","Data":"f8fe942b88750e2c0c546e13f52b710865c9595e05630af90d289a6e045c9f21"}
Sep 08 16:17:24.891395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.891348    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt" podStartSLOduration=2.311959221 podStartE2EDuration="5.891333897s" totalImagesPullingTime="3.579374676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:24.888887738 +0000 UTC m=+274.532964430" watchObservedRunningTime="2026-09-08 16:17:24.891333897 +0000 UTC m=+274.535410630"
Sep 08 16:17:24.914786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:24.914659    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.014643826 podStartE2EDuration="7.91464219s" totalImagesPullingTime="3.88391884s" totalInitContainerRuntime="16.079524ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:24.913078605 +0000 UTC m=+274.557155297" watchObservedRunningTime="2026-09-08 16:17:24.91464219 +0000 UTC m=+274.558718901"
Sep 08 16:17:25.859680 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:25.859646    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:25.860079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:25.859703    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:27.384138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:27.384088    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" podUID="7d4a0352-ac9b-4f54-aed6-b8d56567c0a3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 16:17:27.786381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:27.786353    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:27.786474 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:27.786409    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:27.887368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:27.887342    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"}
Sep 08 16:17:28.893657 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.893620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"}
Sep 08 16:17:28.894042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.893700    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"}
Sep 08 16:17:28.894042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.893740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"}
Sep 08 16:17:28.894042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.893802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"}
Sep 08 16:17:28.894042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.893842    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerStarted","Data":"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"}
Sep 08 16:17:28.922870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:28.922813    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.01516532 podStartE2EDuration="6.922797645s" totalImagesPullingTime="2.887734022s" totalInitContainerRuntime="19.898303ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:28.920548088 +0000 UTC m=+278.564624781" watchObservedRunningTime="2026-09-08 16:17:28.922797645 +0000 UTC m=+278.566874337"
Sep 08 16:17:30.398653 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:30.398618    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:30.499321 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:30.499290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:30.887244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:30.887214    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85fc6fcffb-l6jxt"
Sep 08 16:17:31.492178 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:17:31.492138    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cdhsw" podUID="e6a6631c-8c75-47b9-aec8-2bafa218cc67"
Sep 08 16:17:31.904670 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:31.904639    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:33.329973 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:33.329939    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:17:35.137764 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.137724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:35.140310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.140288    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6a6631c-8c75-47b9-aec8-2bafa218cc67-metrics-tls\") pod \"dns-default-cdhsw\" (UID: \"e6a6631c-8c75-47b9-aec8-2bafa218cc67\") " pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:35.207247 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.207220    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\""
Sep 08 16:17:35.215152 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.215129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:35.306197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.306163    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:35.364375 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.364343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:35.486822 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.486700    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cdhsw"]
Sep 08 16:17:35.489382 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:17:35.489353    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a6631c_8c75_47b9_aec8_2bafa218cc67.slice/crio-429f8f482500c8dca02a7cd5edd6634baa4885f8b632bc19d559e8aeb3a6a95e WatchSource:0}: Error finding container 429f8f482500c8dca02a7cd5edd6634baa4885f8b632bc19d559e8aeb3a6a95e: Status 404 returned error can't find the container with id 429f8f482500c8dca02a7cd5edd6634baa4885f8b632bc19d559e8aeb3a6a95e
Sep 08 16:17:35.857814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.857783    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:35.857964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.857830    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:35.919983 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:35.919949    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdhsw" event={"ID":"e6a6631c-8c75-47b9-aec8-2bafa218cc67","Type":"ContainerStarted","Data":"429f8f482500c8dca02a7cd5edd6634baa4885f8b632bc19d559e8aeb3a6a95e"}
Sep 08 16:17:37.384065 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.384027    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" podUID="7d4a0352-ac9b-4f54-aed6-b8d56567c0a3" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 16:17:37.384513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.384107    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g"
Sep 08 16:17:37.384747 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.384703    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"24b2ce57f8d86584993d833128b75e42f779e4a1dfd6c436847de57c19875bc6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 08 16:17:37.384861 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.384799    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" podUID="7d4a0352-ac9b-4f54-aed6-b8d56567c0a3" containerName="service-proxy" containerID="cri-o://24b2ce57f8d86584993d833128b75e42f779e4a1dfd6c436847de57c19875bc6" gracePeriod=30
Sep 08 16:17:37.786331 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.786300    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:37.786466 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.786349    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:37.931209 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.931133    2571 generic.go:356] "Generic (PLEG): container finished" podID="7d4a0352-ac9b-4f54-aed6-b8d56567c0a3" containerID="24b2ce57f8d86584993d833128b75e42f779e4a1dfd6c436847de57c19875bc6" exitCode=2
Sep 08 16:17:37.931332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.931208    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerDied","Data":"24b2ce57f8d86584993d833128b75e42f779e4a1dfd6c436847de57c19875bc6"}
Sep 08 16:17:37.931332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.931263    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5985dcd66d-dh25g" event={"ID":"7d4a0352-ac9b-4f54-aed6-b8d56567c0a3","Type":"ContainerStarted","Data":"db7cc28c231a45443c0b321bac31fb72109f3510fe29e1120cef2a5975cc1e8d"}
Sep 08 16:17:37.933153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.933131    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdhsw" event={"ID":"e6a6631c-8c75-47b9-aec8-2bafa218cc67","Type":"ContainerStarted","Data":"746590f043aee36968599b937bf985f7456b86289ddacf62daa1cdc898ac55fb"}
Sep 08 16:17:37.933253 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.933169    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cdhsw" event={"ID":"e6a6631c-8c75-47b9-aec8-2bafa218cc67","Type":"ContainerStarted","Data":"2d853a9cc36f11c55887e06b9e86ccd37cf0e18bc462e86e9717fafd9cb0404a"}
Sep 08 16:17:37.933253 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.933198    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:37.967961 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:37.967920    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cdhsw" podStartSLOduration=250.952580232 podStartE2EDuration="4m12.967904986s" totalImagesPullingTime="2.015324754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:37.967667485 +0000 UTC m=+287.611744203" watchObservedRunningTime="2026-09-08 16:17:37.967904986 +0000 UTC m=+287.611981677"
Sep 08 16:17:38.132947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:38.132915    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:38.233395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:38.233323    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:38.899058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:38.899022    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:42.312058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:42.312005    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:45.857936 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.857899    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:45.858287 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.857950    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:45.858287 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.857989    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:45.858455 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.858420    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"e4fb8061dcb11c531056f010bca7f257db40492ea4bac604e2492fff882f9006"} pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 08 16:17:45.861876 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.861849    2571 patch_prober.go:28] interesting pod/image-registry-5789b9fbd6-4kdw6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:45.861971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:45.861892    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:47.938942 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:47.938909    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cdhsw"
Sep 08 16:17:49.075133 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:49.075090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:49.127268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:49.127236    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:49.182563 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:49.182533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:50.773095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:50.773067    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:17:50.773523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:50.773076    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:17:50.777276 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:50.777258    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 16:17:52.714386 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:52.714347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:52.768089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:52.768044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:52.981939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:52.981865    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:53.082610 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:53.082584    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:55.862624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:55.862593    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:17:55.996201 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:55.996162    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:56.096603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:56.096566    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:56.304819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:56.304723    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:17:56.360604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:17:56.360570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:00.400208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:00.400175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:00.500535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:00.500490    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:02.007709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:02.007673    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:05.379270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:05.379234    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:05.479061 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:05.479036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:10.012370 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:10.012333    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqgfn_ded0bbca-b5d7-4dce-80ad-44ce042b3c48/serve-healthcheck-canary/0.log"
Sep 08 16:18:10.878124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:10.878085    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" podUID="9065f588-b363-4690-a1ed-899fbc692584" containerName="registry" containerID="cri-o://e4fb8061dcb11c531056f010bca7f257db40492ea4bac604e2492fff882f9006" gracePeriod=30
Sep 08 16:18:12.059527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:12.059495    2571 generic.go:356] "Generic (PLEG): container finished" podID="9065f588-b363-4690-a1ed-899fbc692584" containerID="e4fb8061dcb11c531056f010bca7f257db40492ea4bac604e2492fff882f9006" exitCode=0
Sep 08 16:18:12.059824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:12.059558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" event={"ID":"9065f588-b363-4690-a1ed-899fbc692584","Type":"ContainerDied","Data":"e4fb8061dcb11c531056f010bca7f257db40492ea4bac604e2492fff882f9006"}
Sep 08 16:18:13.065270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:13.065228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6" event={"ID":"9065f588-b363-4690-a1ed-899fbc692584","Type":"ContainerStarted","Data":"6f148f784d667c7709c8c7871181601e90d500a6c6a8af6b21207b8b6fbaf8ca"}
Sep 08 16:18:13.065664 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:13.065354    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:18:13.324584 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:13.324547    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:13.376874 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:13.376847    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:15.397699 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:15.397660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:15.452329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:15.451920    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:22.730587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:22.730552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:22.796590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:22.796543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:22.846628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:22.846568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:23.330196 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:23.330158    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:23.361520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:23.361495    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:24.124681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:24.124649    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:30.402345 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:30.402306    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:30.457927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:30.457896    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:34.073259 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:34.073223    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5789b9fbd6-4kdw6"
Sep 08 16:18:34.345009 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:34.344927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:34.399684 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:34.399657    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:35.985003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:35.984968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:36.087558 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:36.087521    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:37.044186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044151    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:18:37.044600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044442    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="alertmanager" containerID="cri-o://6847f060db9ff6a8b3c577075306f56fe2ed9f15839b12a8577bd9e5bb2bf352" gracePeriod=120
Sep 08 16:18:37.044600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044500    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-metric" containerID="cri-o://51ae741071cbd15f155f7aa0dc73818ad58d4a2156f1e8fa767c7db3baa3fdc2" gracePeriod=120
Sep 08 16:18:37.044600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044527    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-web" containerID="cri-o://cfe13ef8d3b9adc79e463b15470a174a602ddf02d60ede4468a01de20e5beaa4" gracePeriod=120
Sep 08 16:18:37.044600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044527    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="prom-label-proxy" containerID="cri-o://f8fe942b88750e2c0c546e13f52b710865c9595e05630af90d289a6e045c9f21" gracePeriod=120
Sep 08 16:18:37.044600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.044580    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy" containerID="cri-o://4bfa868b70c19298594d4187a9e2361d59153c2e82025bc484b53aa429cf52a9" gracePeriod=120
Sep 08 16:18:37.045463 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.045121    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="config-reloader" containerID="cri-o://b2ee3caa0159dc701c4b5402603e32c939dc72128b2c5c9d20fd49162210a869" gracePeriod=120
Sep 08 16:18:37.165823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165796    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="f8fe942b88750e2c0c546e13f52b710865c9595e05630af90d289a6e045c9f21" exitCode=0
Sep 08 16:18:37.165823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165820    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="4bfa868b70c19298594d4187a9e2361d59153c2e82025bc484b53aa429cf52a9" exitCode=0
Sep 08 16:18:37.165823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165831    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="b2ee3caa0159dc701c4b5402603e32c939dc72128b2c5c9d20fd49162210a869" exitCode=0
Sep 08 16:18:37.165990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165839    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="6847f060db9ff6a8b3c577075306f56fe2ed9f15839b12a8577bd9e5bb2bf352" exitCode=0
Sep 08 16:18:37.165990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165865    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"f8fe942b88750e2c0c546e13f52b710865c9595e05630af90d289a6e045c9f21"}
Sep 08 16:18:37.165990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165930    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"4bfa868b70c19298594d4187a9e2361d59153c2e82025bc484b53aa429cf52a9"}
Sep 08 16:18:37.165990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"b2ee3caa0159dc701c4b5402603e32c939dc72128b2c5c9d20fd49162210a869"}
Sep 08 16:18:37.165990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:37.165980    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"6847f060db9ff6a8b3c577075306f56fe2ed9f15839b12a8577bd9e5bb2bf352"}
Sep 08 16:18:38.173909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.173883    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="51ae741071cbd15f155f7aa0dc73818ad58d4a2156f1e8fa767c7db3baa3fdc2" exitCode=0
Sep 08 16:18:38.173909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.173907    2571 generic.go:356] "Generic (PLEG): container finished" podID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerID="cfe13ef8d3b9adc79e463b15470a174a602ddf02d60ede4468a01de20e5beaa4" exitCode=0
Sep 08 16:18:38.174230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.173939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"51ae741071cbd15f155f7aa0dc73818ad58d4a2156f1e8fa767c7db3baa3fdc2"}
Sep 08 16:18:38.174230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.174013    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"cfe13ef8d3b9adc79e463b15470a174a602ddf02d60ede4468a01de20e5beaa4"}
Sep 08 16:18:38.286167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.286142    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:18:38.328150 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328118    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328168    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328195    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328215    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328239    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328265    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328289    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328358    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328383    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328411    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crt6q\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328494    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328520    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web\") pod \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\" (UID: \"f23cafd0-e6ad-45fb-a806-2578acbbdefb\") "
Sep 08 16:18:38.328970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328619    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:38.328970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328632    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:38.328970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328839    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-metrics-client-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.328970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.328861    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.330837 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.330758    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335041    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335124    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335176    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out" (OuterVolumeSpecName: "config-out") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335255    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335333    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume" (OuterVolumeSpecName: "config-volume") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335553    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.336024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.335646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.336450 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.336070    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q" (OuterVolumeSpecName: "kube-api-access-crt6q") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "kube-api-access-crt6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:38.338860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.338742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.347879 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.347849    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config" (OuterVolumeSpecName: "web-config") pod "f23cafd0-e6ad-45fb-a806-2578acbbdefb" (UID: "f23cafd0-e6ad-45fb-a806-2578acbbdefb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:38.429864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429831    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.429864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429853    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.429864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429865    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-out\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429874    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429883    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-tls-assets\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429893    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f23cafd0-e6ad-45fb-a806-2578acbbdefb-alertmanager-main-db\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429901    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-secret-alertmanager-main-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429912    2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429919    2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-cluster-tls-config\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429927    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f23cafd0-e6ad-45fb-a806-2578acbbdefb-web-config\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:38.430052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:38.429936    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crt6q\" (UniqueName: \"kubernetes.io/projected/f23cafd0-e6ad-45fb-a806-2578acbbdefb-kube-api-access-crt6q\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:39.182007 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.181935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f23cafd0-e6ad-45fb-a806-2578acbbdefb","Type":"ContainerDied","Data":"dda21a31e49f63e9d2ed49f87f3fefe2f8ff7c8d917f6d18a4699d371a1a7b0c"}
Sep 08 16:18:39.182007 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.181992    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 16:18:39.182410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.182035    2571 scope.go:117] "RemoveContainer" containerID="f8fe942b88750e2c0c546e13f52b710865c9595e05630af90d289a6e045c9f21"
Sep 08 16:18:39.189796 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.189778    2571 scope.go:117] "RemoveContainer" containerID="51ae741071cbd15f155f7aa0dc73818ad58d4a2156f1e8fa767c7db3baa3fdc2"
Sep 08 16:18:39.197140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.197122    2571 scope.go:117] "RemoveContainer" containerID="4bfa868b70c19298594d4187a9e2361d59153c2e82025bc484b53aa429cf52a9"
Sep 08 16:18:39.200875 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.200817    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:18:39.205247 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.205222    2571 scope.go:117] "RemoveContainer" containerID="cfe13ef8d3b9adc79e463b15470a174a602ddf02d60ede4468a01de20e5beaa4"
Sep 08 16:18:39.205792 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.205756    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 16:18:39.214688 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.214671    2571 scope.go:117] "RemoveContainer" containerID="b2ee3caa0159dc701c4b5402603e32c939dc72128b2c5c9d20fd49162210a869"
Sep 08 16:18:39.221556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.221540    2571 scope.go:117] "RemoveContainer" containerID="6847f060db9ff6a8b3c577075306f56fe2ed9f15839b12a8577bd9e5bb2bf352"
Sep 08 16:18:39.228907 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:39.228890    2571 scope.go:117] "RemoveContainer" containerID="bb11f1083f5cba342e2d5e1a6f5ee35db814f91cb595f99663c8234c01c5da79"
Sep 08 16:18:40.886498 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:40.886459    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" path="/var/lib/kubelet/pods/f23cafd0-e6ad-45fb-a806-2578acbbdefb/volumes"
Sep 08 16:18:41.339504 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339463    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:41.339866 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339834    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="prometheus" containerID="cri-o://1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f" gracePeriod=600
Sep 08 16:18:41.340002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339860    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-web" containerID="cri-o://c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61" gracePeriod=600
Sep 08 16:18:41.340002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339865    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="config-reloader" containerID="cri-o://9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f" gracePeriod=600
Sep 08 16:18:41.340002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339883    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f" gracePeriod=600
Sep 08 16:18:41.340002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339856    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="thanos-sidecar" containerID="cri-o://48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218" gracePeriod=600
Sep 08 16:18:41.340197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:41.339876    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy" containerID="cri-o://14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362" gracePeriod=600
Sep 08 16:18:42.198629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198596    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f" exitCode=0
Sep 08 16:18:42.198629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198621    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362" exitCode=0
Sep 08 16:18:42.198629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198628    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218" exitCode=0
Sep 08 16:18:42.198629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198635    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f" exitCode=0
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198641    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f" exitCode=0
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198676    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"}
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198739    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"}
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198783    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"}
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198813    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"}
Sep 08 16:18:42.199108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.198836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"}
Sep 08 16:18:42.578153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.578130    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:42.657791 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657748    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.657913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657797    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.657913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657823    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.657913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657845    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.657913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657878    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657924    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657954    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.657980    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658010    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpnf7\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658033    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.658965 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658083    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659266 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658236    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658290    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658293    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.658896    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659232    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659248    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659337    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659362    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659375    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659403    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659459    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659484    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.659673 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659516    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config\") pod \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\" (UID: \"383a834b-34f2-4dfb-97b7-a03f8ff21b5e\") "
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659762    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-db\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659813    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659828    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659842    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659857    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.659896    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.660897 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.660868    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.660999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.660943    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.662055 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.662014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.662229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.662195    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.663041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.662995    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:42.663884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.663793    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.663884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.663850    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7" (OuterVolumeSpecName: "kube-api-access-tpnf7") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "kube-api-access-tpnf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:42.664035 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.663960    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.664506 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.664467    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.664594 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.664518    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out" (OuterVolumeSpecName: "config-out") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:18:42.665297 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.665267    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config" (OuterVolumeSpecName: "config") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.665638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.665613    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:42.665730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.665700    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.671681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.671654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config" (OuterVolumeSpecName: "web-config") pod "383a834b-34f2-4dfb-97b7-a03f8ff21b5e" (UID: "383a834b-34f2-4dfb-97b7-a03f8ff21b5e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:42.760277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760218    2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-metrics-client-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760241    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-web-config\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760256    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnf7\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-kube-api-access-tpnf7\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760268    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-tls-assets\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760280    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config-out\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760292    2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760307    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760320    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760334    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-configmap-metrics-client-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760347    2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-grpc-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760360    2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-kube-rbac-proxy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760372    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760385    2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-config\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:42.760489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:42.760397    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/383a834b-34f2-4dfb-97b7-a03f8ff21b5e-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:18:43.205267 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.205241    2571 generic.go:356] "Generic (PLEG): container finished" podID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerID="c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61" exitCode=0
Sep 08 16:18:43.205629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.205308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"}
Sep 08 16:18:43.205629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.205318    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.205629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.205362    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"383a834b-34f2-4dfb-97b7-a03f8ff21b5e","Type":"ContainerDied","Data":"fade843eb014e1ce8e5dc46903426144f7a42595157b98d508c3ee38ba7cf645"}
Sep 08 16:18:43.205629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.205393    2571 scope.go:117] "RemoveContainer" containerID="5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"
Sep 08 16:18:43.213864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.213842    2571 scope.go:117] "RemoveContainer" containerID="14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"
Sep 08 16:18:43.220678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.220663    2571 scope.go:117] "RemoveContainer" containerID="c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"
Sep 08 16:18:43.227573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.227545    2571 scope.go:117] "RemoveContainer" containerID="48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"
Sep 08 16:18:43.228481 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.228460    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:43.233169 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.233150    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:43.235048 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.235030    2571 scope.go:117] "RemoveContainer" containerID="9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"
Sep 08 16:18:43.243263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.243238    2571 scope.go:117] "RemoveContainer" containerID="1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"
Sep 08 16:18:43.250880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.250862    2571 scope.go:117] "RemoveContainer" containerID="259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3"
Sep 08 16:18:43.257330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.257314    2571 scope.go:117] "RemoveContainer" containerID="5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"
Sep 08 16:18:43.257577 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.257558    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f\": container with ID starting with 5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f not found: ID does not exist" containerID="5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"
Sep 08 16:18:43.257622 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.257586    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f"} err="failed to get container status \"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f\": rpc error: code = NotFound desc = could not find container \"5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f\": container with ID starting with 5032f4ed110c4ddcf8901f93bd0fb182d10606e683053b383f2cefe8e79a4f1f not found: ID does not exist"
Sep 08 16:18:43.257622 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.257605    2571 scope.go:117] "RemoveContainer" containerID="14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"
Sep 08 16:18:43.257886 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.257841    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362\": container with ID starting with 14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362 not found: ID does not exist" containerID="14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"
Sep 08 16:18:43.257886 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.257871    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362"} err="failed to get container status \"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362\": rpc error: code = NotFound desc = could not find container \"14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362\": container with ID starting with 14f5de1edf7e1e3a28ec1e7832cd003a8c62e2d0de75de71e5b0ab7404ba7362 not found: ID does not exist"
Sep 08 16:18:43.258018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.257892    2571 scope.go:117] "RemoveContainer" containerID="c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"
Sep 08 16:18:43.258289 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.258258    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61\": container with ID starting with c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61 not found: ID does not exist" containerID="c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"
Sep 08 16:18:43.258367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.258298    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61"} err="failed to get container status \"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61\": rpc error: code = NotFound desc = could not find container \"c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61\": container with ID starting with c3aaace39035e4d124d0e9a90618a276167cb8bf1a36f120f387a4ef97b65b61 not found: ID does not exist"
Sep 08 16:18:43.258367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.258337    2571 scope.go:117] "RemoveContainer" containerID="48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"
Sep 08 16:18:43.258919 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.258895    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218\": container with ID starting with 48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218 not found: ID does not exist" containerID="48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"
Sep 08 16:18:43.259003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.258928    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218"} err="failed to get container status \"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218\": rpc error: code = NotFound desc = could not find container \"48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218\": container with ID starting with 48fb5ae293c812a23e5846e02340b1c2ca314d0bfb6fe1f55cd79c478d988218 not found: ID does not exist"
Sep 08 16:18:43.259003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.258949    2571 scope.go:117] "RemoveContainer" containerID="9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"
Sep 08 16:18:43.259263 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.259245    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f\": container with ID starting with 9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f not found: ID does not exist" containerID="9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"
Sep 08 16:18:43.259323 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259269    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f"} err="failed to get container status \"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f\": rpc error: code = NotFound desc = could not find container \"9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f\": container with ID starting with 9213deec15e9573e5ff30985c87c885987f3680442f883d5dd98c320e2a2239f not found: ID does not exist"
Sep 08 16:18:43.259323 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259306    2571 scope.go:117] "RemoveContainer" containerID="1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"
Sep 08 16:18:43.259599 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.259584    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f\": container with ID starting with 1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f not found: ID does not exist" containerID="1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"
Sep 08 16:18:43.259641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259603    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f"} err="failed to get container status \"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f\": rpc error: code = NotFound desc = could not find container \"1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f\": container with ID starting with 1bdb4ca6eec19ced2e6fd8a0f93e669b4225278de7fe836d6b3ae2ff73a6af6f not found: ID does not exist"
Sep 08 16:18:43.259641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259618    2571 scope.go:117] "RemoveContainer" containerID="259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3"
Sep 08 16:18:43.259866 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259850    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:43.259918 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:18:43.259875    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3\": container with ID starting with 259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3 not found: ID does not exist" containerID="259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3"
Sep 08 16:18:43.259918 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.259895    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3"} err="failed to get container status \"259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3\": rpc error: code = NotFound desc = could not find container \"259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3\": container with ID starting with 259e40b117b46c2b6d1cfa4a3b8578536a5f5b5c8c694dc9c745fbdfee64acb3 not found: ID does not exist"
Sep 08 16:18:43.260088 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260077    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="prometheus"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260089    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="prometheus"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260101    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="init-config-reloader"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260106    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="init-config-reloader"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260114    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260119    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.260127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260124    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="prom-label-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260129    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="prom-label-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260138    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="init-config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260143    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="init-config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260149    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260155    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260161    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="thanos-sidecar"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260166    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="thanos-sidecar"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260174    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="alertmanager"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260179    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="alertmanager"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260185    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260190    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260196    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260200    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260206    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-thanos"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260211    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-thanos"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260219    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260224    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260229    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260235    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260240    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-metric"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260245    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-metric"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260282    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="alertmanager"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260290    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260296    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-thanos"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260302    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260307    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="config-reloader"
Sep 08 16:18:43.260301 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260311    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-metric"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260317    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="prom-label-proxy"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260323    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260330    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="thanos-sidecar"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260335    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cafd0-e6ad-45fb-a806-2578acbbdefb" containerName="kube-rbac-proxy"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260340    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="prometheus"
Sep 08 16:18:43.261342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.260343    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" containerName="kube-rbac-proxy-web"
Sep 08 16:18:43.268183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.268161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.270870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.270851    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 16:18:43.271117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271099    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 16:18:43.271226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271118    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 16:18:43.271226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271128    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 16:18:43.271226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271105    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 16:18:43.271226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271204    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4n79w\""
Sep 08 16:18:43.271444 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271235    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-731mbug06c1v\""
Sep 08 16:18:43.271499 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271472    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 16:18:43.271553 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271526    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 16:18:43.271603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271551    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 16:18:43.271936 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271921    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 16:18:43.272012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271934    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 16:18:43.272012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.271969    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 16:18:43.272124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.272030    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 16:18:43.272385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.272366    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 16:18:43.273538 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.273520    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 16:18:43.277436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.277418    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 16:18:43.279828 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.279810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:43.366453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvhl\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-kube-api-access-7fvhl\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366621    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366702    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366805    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config-out\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.366890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.367219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366909    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.367219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.367219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.366976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.367219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.367003    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.367219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.367024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-web-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvhl\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-kube-api-access-7fvhl\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config-out\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468648    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.468735 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-web-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.468864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.469023    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.469092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469615 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.469501    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.469615 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.469549    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.470384 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.470353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.471645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.471620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.471739 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.471658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.471831 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.471796    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.472282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.471984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.472282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.472217    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.472591 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.472467    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.472591 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.472504    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.473350 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.473320    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.473431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.473389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.473865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.473841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/368af9e1-f4c3-456e-9ada-ccc8f53dad72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.474426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.474394    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config-out\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.474517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.474494    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.475101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.475082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.475153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.475105    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/368af9e1-f4c3-456e-9ada-ccc8f53dad72-web-config\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.476424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.476405    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvhl\" (UniqueName: \"kubernetes.io/projected/368af9e1-f4c3-456e-9ada-ccc8f53dad72-kube-api-access-7fvhl\") pod \"prometheus-k8s-0\" (UID: \"368af9e1-f4c3-456e-9ada-ccc8f53dad72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.578028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.578002    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:43.868393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:43.868364    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 16:18:43.875363 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:18:43.875340    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368af9e1_f4c3_456e_9ada_ccc8f53dad72.slice/crio-47db47c8804b2f8b66cb4136e80371f54a7896313560f09d3ba666b90fca53ca WatchSource:0}: Error finding container 47db47c8804b2f8b66cb4136e80371f54a7896313560f09d3ba666b90fca53ca: Status 404 returned error can't find the container with id 47db47c8804b2f8b66cb4136e80371f54a7896313560f09d3ba666b90fca53ca
Sep 08 16:18:44.211911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.211840    2571 generic.go:356] "Generic (PLEG): container finished" podID="368af9e1-f4c3-456e-9ada-ccc8f53dad72" containerID="2466ea9045391eba03e947e97b6af719ecb07257327ddc0bd01ab46d30706ed2" exitCode=0
Sep 08 16:18:44.211911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.211905    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerDied","Data":"2466ea9045391eba03e947e97b6af719ecb07257327ddc0bd01ab46d30706ed2"}
Sep 08 16:18:44.212249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.211953    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"47db47c8804b2f8b66cb4136e80371f54a7896313560f09d3ba666b90fca53ca"}
Sep 08 16:18:44.339539 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.339509    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:44.444241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.444139    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:44.540005 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.539979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:44.887940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:44.887909    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383a834b-34f2-4dfb-97b7-a03f8ff21b5e" path="/var/lib/kubelet/pods/383a834b-34f2-4dfb-97b7-a03f8ff21b5e/volumes"
Sep 08 16:18:45.072838 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.072793    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:45.127116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.127091    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:45.218927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.218859    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"bf80446b0672561bbc42eee09a1472e433abfb86503efa8434070cad69cc3835"}
Sep 08 16:18:45.219358 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.219326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"2efb8c59932410f9716f0649462ff01b05a2b4bc7d22fcb4f7f86cb47efed73f"}
Sep 08 16:18:45.219452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.219383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"7a2bd0520b9dba14259e532906a1116e0cb9900763c1fc29fa79238d5e53f8ba"}
Sep 08 16:18:45.219452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.219424    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"d2c29dcd1d97ec9383c3cc0369dea0b4ce25ab8bd7fe255f19850b2c4f142340"}
Sep 08 16:18:45.219552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.219462    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"9c362a6c6fe824394c8c6d71de0b6e16576dd8763b692fd228c56ca9716e91d2"}
Sep 08 16:18:45.219552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.219502    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"368af9e1-f4c3-456e-9ada-ccc8f53dad72","Type":"ContainerStarted","Data":"8797921996d623c8ff1f808edb0fe091817de61cd5f6ffa4c10f2b2d4d65a79a"}
Sep 08 16:18:45.247630 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.247584    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.230945346 podStartE2EDuration="2.247565094s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.619748ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:45.246296559 +0000 UTC m=+354.890373252" watchObservedRunningTime="2026-09-08 16:18:45.247565094 +0000 UTC m=+354.891641786"
Sep 08 16:18:45.408913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.408888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:45.466719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:45.466681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:48.578613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:48.578578    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:18:54.952546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:54.952505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:55.056803 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:55.056746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:18:55.154131 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:18:55.154102    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:19:39.943944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:39.943880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:19:43.578524 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:43.578481    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:19:43.595545 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:43.595516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:19:44.452594 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:44.452565    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 16:19:51.209032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:51.208997    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:19:51.266710 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:19:51.266659    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:03.116018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:03.115979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:03.252441 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:03.252399    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:15.405141 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:15.405020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:15.505590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:15.505552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:15.750114 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:15.750026    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:20.487432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:20.487395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:20.546597 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:20.546559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:31.756052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:31.756010    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:31.856471 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:31.856432    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:31.917007 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:31.916976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:37.820326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:37.820290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:37.882080 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:37.882043    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:20:50.636496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:20:50.636455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:03.433373 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:03.433334    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:03.531325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:03.531288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:25.117054 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:25.117009    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:35.172132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:35.172100    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:35.992817 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:35.992787    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:38.795783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:38.795741    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:38.894398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:38.894367    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:40.339482 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:40.339450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:40.441913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:40.441881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:40.965597 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:40.965563    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:41.077197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:41.077161    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:43.187522 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:43.187435    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:43.365727 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:43.365697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:45.583211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:45.583175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:46.933380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:46.933343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:48.359370 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:48.359329    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:48.459605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:48.459568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:51.563327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:51.563278    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:53.068063 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:53.068019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:53.164581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:53.164548    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:55.578869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:55.578828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:58.615286 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:58.615255    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:21:58.674572 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:21:58.674547    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:01.014780 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:01.014738    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:17.224622 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:17.224584    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:17.317105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:17.317057    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:17.871446 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:17.871411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:18.114953 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.114916    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:18.244566 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.244484    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"]
Sep 08 16:22:18.247861 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.247831    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.250494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.250467    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 16:22:18.250607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.250591    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 16:22:18.250757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.250741    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qjv57\""
Sep 08 16:22:18.251019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.251004    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 16:22:18.251089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.251066    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 16:22:18.266238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.266214    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"]
Sep 08 16:22:18.306612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.306575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.306612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.306608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfgz\" (UniqueName: \"kubernetes.io/projected/ddb04eed-c133-41be-bcc1-09386a1b81b3-kube-api-access-mlfgz\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.306871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.306635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.306871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.306698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.306871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.306745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408081 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfgz\" (UniqueName: \"kubernetes.io/projected/ddb04eed-c133-41be-bcc1-09386a1b81b3-kube-api-access-mlfgz\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408173    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.408484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.408457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.410687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.410665    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04eed-c133-41be-bcc1-09386a1b81b3-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.410958 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.410940    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.411028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.410963    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddb04eed-c133-41be-bcc1-09386a1b81b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.419389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.419363    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfgz\" (UniqueName: \"kubernetes.io/projected/ddb04eed-c133-41be-bcc1-09386a1b81b3-kube-api-access-mlfgz\") pod \"opendatahub-operator-controller-manager-68b89c556d-jfb2p\" (UID: \"ddb04eed-c133-41be-bcc1-09386a1b81b3\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.557622 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.557526    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:18.743213 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.743182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:18.823162 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.823130    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:18.836902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.836874    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"]
Sep 08 16:22:18.839722 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:22:18.839694    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb04eed_c133_41be_bcc1_09386a1b81b3.slice/crio-94ea7f986e5a9d58900a4c4580d05457e4fc7e505393281b08d6cb01e27e87a0 WatchSource:0}: Error finding container 94ea7f986e5a9d58900a4c4580d05457e4fc7e505393281b08d6cb01e27e87a0: Status 404 returned error can't find the container with id 94ea7f986e5a9d58900a4c4580d05457e4fc7e505393281b08d6cb01e27e87a0
Sep 08 16:22:18.841442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.841422    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:22:18.995124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:18.995092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p" event={"ID":"ddb04eed-c133-41be-bcc1-09386a1b81b3","Type":"ContainerStarted","Data":"94ea7f986e5a9d58900a4c4580d05457e4fc7e505393281b08d6cb01e27e87a0"}
Sep 08 16:22:23.013596 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:23.013564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p" event={"ID":"ddb04eed-c133-41be-bcc1-09386a1b81b3","Type":"ContainerStarted","Data":"9c9eab71f64bffaa024e77a6dcb2d36b1326fa34cab56d3cfbe2e5df549b8f34"}
Sep 08 16:22:23.288527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:23.288439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:23.396344 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:23.396303    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:24.019110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:24.019077    2571 generic.go:356] "Generic (PLEG): container finished" podID="ddb04eed-c133-41be-bcc1-09386a1b81b3" containerID="9c9eab71f64bffaa024e77a6dcb2d36b1326fa34cab56d3cfbe2e5df549b8f34" exitCode=0
Sep 08 16:22:24.019544 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:24.019141    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p" event={"ID":"ddb04eed-c133-41be-bcc1-09386a1b81b3","Type":"ContainerDied","Data":"9c9eab71f64bffaa024e77a6dcb2d36b1326fa34cab56d3cfbe2e5df549b8f34"}
Sep 08 16:22:25.029230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:25.029200    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p" event={"ID":"ddb04eed-c133-41be-bcc1-09386a1b81b3","Type":"ContainerStarted","Data":"34a3234e19cebbcae18104e65d599cfa69d3de8edd8acdeaf486ec72d2715827"}
Sep 08 16:22:25.029598 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:25.029323    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:25.050964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:25.050911    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p" podStartSLOduration=2.578907407 podStartE2EDuration="7.050892304s" totalImagesPullingTime="3.414200335s" totalInitContainerRuntime="1.057784562s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:25.050469552 +0000 UTC m=+574.694546244" watchObservedRunningTime="2026-09-08 16:22:25.050892304 +0000 UTC m=+574.694968994"
Sep 08 16:22:30.226118 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:30.226085    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:30.325068 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:30.325026    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:36.035464 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:36.035425    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jfb2p"
Sep 08 16:22:37.575685 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:37.575652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:37.693293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:37.693251    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:39.181408 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:39.181373    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:39.247992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:39.247963    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:41.761561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:41.761520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:41.862279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:41.862243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:43.175906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:43.175868    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:43.405003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:43.404966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:43.509514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:43.509427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:43.561343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:43.561298    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:44.238786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:44.238733    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:44.298603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:44.298569    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:45.082706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:45.082668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.474705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.474670    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.535535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.535499    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.636178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.636144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.736191 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.736112    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.838405 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.838364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:48.995326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:48.995239    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:49.252513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:49.252418    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:49.519124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:49.519024    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:49.724978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:49.724942    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:49.819514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:49.819481    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:49.919557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:49.919519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:50.020079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:50.020051    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:50.119972 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:50.119877    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:50.804022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:50.803998    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:22:50.804429 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:50.804239    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:22:56.478325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.478286    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"]
Sep 08 16:22:56.481841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.481825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.485549 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485522    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 16:22:56.485645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485522    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 16:22:56.485645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485576    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 16:22:56.485645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485633    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 16:22:56.485645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485523    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:22:56.485838 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.485821    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zq85p\""
Sep 08 16:22:56.490902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.490882    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"]
Sep 08 16:22:56.604105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.604071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.604305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.604117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6shb\" (UniqueName: \"kubernetes.io/projected/446d93c8-7319-4872-9910-ba31fb9c1703-kube-api-access-l6shb\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.604305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.604151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-metrics-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.604305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.604251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/446d93c8-7319-4872-9910-ba31fb9c1703-manager-config\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.705049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.705012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/446d93c8-7319-4872-9910-ba31fb9c1703-manager-config\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.705245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.705062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.705245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.705197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6shb\" (UniqueName: \"kubernetes.io/projected/446d93c8-7319-4872-9910-ba31fb9c1703-kube-api-access-l6shb\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.705360 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.705248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-metrics-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.705800 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.705750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/446d93c8-7319-4872-9910-ba31fb9c1703-manager-config\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.707802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.707757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.707904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.707760    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/446d93c8-7319-4872-9910-ba31fb9c1703-metrics-cert\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.728912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.728842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6shb\" (UniqueName: \"kubernetes.io/projected/446d93c8-7319-4872-9910-ba31fb9c1703-kube-api-access-l6shb\") pod \"lws-controller-manager-78fd8475-d8w8l\" (UID: \"446d93c8-7319-4872-9910-ba31fb9c1703\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:56.791255 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:56.791222    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:22:57.088873 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:57.088842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"]
Sep 08 16:22:57.095424 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:22:57.095392    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446d93c8_7319_4872_9910_ba31fb9c1703.slice/crio-a39f2151b558a0fa766309b33e3deceeb01650b23fbeefaec553ba4b62f83f75 WatchSource:0}: Error finding container a39f2151b558a0fa766309b33e3deceeb01650b23fbeefaec553ba4b62f83f75: Status 404 returned error can't find the container with id a39f2151b558a0fa766309b33e3deceeb01650b23fbeefaec553ba4b62f83f75
Sep 08 16:22:57.151105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:57.151080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l" event={"ID":"446d93c8-7319-4872-9910-ba31fb9c1703","Type":"ContainerStarted","Data":"a39f2151b558a0fa766309b33e3deceeb01650b23fbeefaec553ba4b62f83f75"}
Sep 08 16:22:59.148916 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.148876    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:59.214614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.214573    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"]
Sep 08 16:22:59.218430 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.218403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.221392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.221364    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-525g5\""
Sep 08 16:22:59.221647 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.221622    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 08 16:22:59.233440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.233381    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"]
Sep 08 16:22:59.265221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.265178    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:59.331884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.331848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflwk\" (UniqueName: \"kubernetes.io/projected/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-kube-api-access-hflwk\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.332065 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.331904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.360515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.360463    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:59.432506 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.432430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflwk\" (UniqueName: \"kubernetes.io/projected/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-kube-api-access-hflwk\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.432506 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.432501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.432720 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:22:59.432627    2571 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 08 16:22:59.432720 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:22:59.432688    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert podName:cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c nodeName:}" failed. No retries permitted until 2026-09-08 16:22:59.932671599 +0000 UTC m=+609.576748270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert") pod "kserve-controller-manager-d6f4b65b8-cztqr" (UID: "cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c") : secret "kserve-webhook-server-cert" not found
Sep 08 16:22:59.453190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.453164    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflwk\" (UniqueName: \"kubernetes.io/projected/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-kube-api-access-hflwk\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.750809 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.750682    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:22:59.937943 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.937900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:22:59.940499 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:22:59.940480    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cztqr\" (UID: \"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:23:00.129834 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.129800    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:23:00.168110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.167859    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l" event={"ID":"446d93c8-7319-4872-9910-ba31fb9c1703","Type":"ContainerStarted","Data":"d16cb3f36a1565dd074b8ad35cc75cd881ddefff709cb6902f8c86794cc1c757"}
Sep 08 16:23:00.168552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.168213    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:23:00.186834 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.186755    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l" podStartSLOduration=1.7892752079999998 podStartE2EDuration="4.186732718s" totalImagesPullingTime="2.39745751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:00.185756662 +0000 UTC m=+609.829833355" watchObservedRunningTime="2026-09-08 16:23:00.186732718 +0000 UTC m=+609.830809408"
Sep 08 16:23:00.407316 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.407233    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"]
Sep 08 16:23:00.410634 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:23:00.410607    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcf1f8f_8ed4_465b_b7a5_57b6dee6749c.slice/crio-1c22c2875aec6a78ff83ea62aacfb2142a9e5491887a40c4a16c2ca7ed58431e WatchSource:0}: Error finding container 1c22c2875aec6a78ff83ea62aacfb2142a9e5491887a40c4a16c2ca7ed58431e: Status 404 returned error can't find the container with id 1c22c2875aec6a78ff83ea62aacfb2142a9e5491887a40c4a16c2ca7ed58431e
Sep 08 16:23:00.460365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.460324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:23:00.607471 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.607439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:23:00.715179 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:00.715058    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:23:01.173083 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:01.173046    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr" event={"ID":"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c","Type":"ContainerStarted","Data":"1c22c2875aec6a78ff83ea62aacfb2142a9e5491887a40c4a16c2ca7ed58431e"}
Sep 08 16:23:03.183520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:03.183489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr" event={"ID":"cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c","Type":"ContainerStarted","Data":"d87d793d0c8989fe649df6f041fa36e051f09ec6bdf68f3a8bd3111717937641"}
Sep 08 16:23:03.183917 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:03.183606    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:23:03.201676 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:03.201631    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr" podStartSLOduration=1.5793915859999998 podStartE2EDuration="4.201616372s" totalImagesPullingTime="2.622224786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:03.200241166 +0000 UTC m=+612.844317866" watchObservedRunningTime="2026-09-08 16:23:03.201616372 +0000 UTC m=+612.845693064"
Sep 08 16:23:11.175339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:11.175305    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78fd8475-d8w8l"
Sep 08 16:23:34.192459 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:34.192421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cztqr"
Sep 08 16:23:49.589248 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:49.589210    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:23:49.683246 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:23:49.683196    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:05.504847 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:05.504813    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:07.909190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.909153    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"]
Sep 08 16:24:07.911442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.911423    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:07.914015 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.913991    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bm6tl\""
Sep 08 16:24:07.914886 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.914860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 16:24:07.914985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.914924    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 16:24:07.930062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:07.930035    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"]
Sep 08 16:24:08.009929 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.009899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvvz\" (UniqueName: \"kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz\") pod \"limitador-operator-controller-manager-74495c69f8-n8bd7\" (UID: \"9140e2ba-ec45-471f-9f83-4a9dec4cca44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:08.111022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.110994    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvvz\" (UniqueName: \"kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz\") pod \"limitador-operator-controller-manager-74495c69f8-n8bd7\" (UID: \"9140e2ba-ec45-471f-9f83-4a9dec4cca44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:08.129183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.129159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvvz\" (UniqueName: \"kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz\") pod \"limitador-operator-controller-manager-74495c69f8-n8bd7\" (UID: \"9140e2ba-ec45-471f-9f83-4a9dec4cca44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:08.221684 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.221619    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:08.447576 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.447543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:08.495312 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:08.495209    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"]
Sep 08 16:24:08.497906 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:24:08.497881    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9140e2ba_ec45_471f_9f83_4a9dec4cca44.slice/crio-255bf28234b6e3df5d049d019ac2d0a695cfd8b67c39fe2f713951931d9ad0de WatchSource:0}: Error finding container 255bf28234b6e3df5d049d019ac2d0a695cfd8b67c39fe2f713951931d9ad0de: Status 404 returned error can't find the container with id 255bf28234b6e3df5d049d019ac2d0a695cfd8b67c39fe2f713951931d9ad0de
Sep 08 16:24:09.038034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:09.037998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:09.451809 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:09.451726    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7" event={"ID":"9140e2ba-ec45-471f-9f83-4a9dec4cca44","Type":"ContainerStarted","Data":"255bf28234b6e3df5d049d019ac2d0a695cfd8b67c39fe2f713951931d9ad0de"}
Sep 08 16:24:11.464204 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:11.464165    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7" event={"ID":"9140e2ba-ec45-471f-9f83-4a9dec4cca44","Type":"ContainerStarted","Data":"54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000"}
Sep 08 16:24:11.464588 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:11.464308    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:11.491947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:11.491895    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7" podStartSLOduration=1.975153493 podStartE2EDuration="4.491879377s" totalImagesPullingTime="2.516725884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:11.491584695 +0000 UTC m=+681.135661388" watchObservedRunningTime="2026-09-08 16:24:11.491879377 +0000 UTC m=+681.135956068"
Sep 08 16:24:12.637671 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:12.637634    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:14.278083 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:14.278048    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:14.885213 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:14.885169    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:15.278089 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:15.277984    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:15.575105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:15.575074    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:15.671423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:15.671394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:16.913434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:16.913397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:17.915382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:17.915316    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:20.044418 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:20.044376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:20.148151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:20.148106    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:22.470295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:22.470267    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:23.873709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:23.873676    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:23.972065 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:23.972026    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:24.076484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:24.076450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:24.175995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:24.175893    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:24.771645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:24.771606    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:25.425262 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.425232    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"]
Sep 08 16:24:25.428280 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.428260    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.430862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.430837    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5p6kn\""
Sep 08 16:24:25.430969 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.430858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 16:24:25.431138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.431124    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 16:24:25.436847 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.436821    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"]
Sep 08 16:24:25.491917 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.491873    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:25.559932 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.559892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljhm\" (UniqueName: \"kubernetes.io/projected/293cbeeb-1f98-40f1-8776-f613368c1b92-kube-api-access-4ljhm\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.560097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.559945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293cbeeb-1f98-40f1-8776-f613368c1b92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.560097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.560009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293cbeeb-1f98-40f1-8776-f613368c1b92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.637714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.637662    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:25.661368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.661338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293cbeeb-1f98-40f1-8776-f613368c1b92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.661468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.661430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljhm\" (UniqueName: \"kubernetes.io/projected/293cbeeb-1f98-40f1-8776-f613368c1b92-kube-api-access-4ljhm\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.661468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.661461    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293cbeeb-1f98-40f1-8776-f613368c1b92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.662144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.662121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293cbeeb-1f98-40f1-8776-f613368c1b92-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.664034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.664003    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293cbeeb-1f98-40f1-8776-f613368c1b92-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.669964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.669940    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljhm\" (UniqueName: \"kubernetes.io/projected/293cbeeb-1f98-40f1-8776-f613368c1b92-kube-api-access-4ljhm\") pod \"kuadrant-console-plugin-fbdd4f445-zdzzj\" (UID: \"293cbeeb-1f98-40f1-8776-f613368c1b92\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:25.738146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:25.738059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"
Sep 08 16:24:26.015398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:26.015373    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj"]
Sep 08 16:24:26.017874 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:24:26.017845    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod293cbeeb_1f98_40f1_8776_f613368c1b92.slice/crio-c15da82016212f7a826c40accd1e380b53445631e51f0e720aa8d36b2c209a64 WatchSource:0}: Error finding container c15da82016212f7a826c40accd1e380b53445631e51f0e720aa8d36b2c209a64: Status 404 returned error can't find the container with id c15da82016212f7a826c40accd1e380b53445631e51f0e720aa8d36b2c209a64
Sep 08 16:24:26.531595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:26.531557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj" event={"ID":"293cbeeb-1f98-40f1-8776-f613368c1b92","Type":"ContainerStarted","Data":"c15da82016212f7a826c40accd1e380b53445631e51f0e720aa8d36b2c209a64"}
Sep 08 16:24:26.833687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:26.833653    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:26.941185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:26.941144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:31.554934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:31.554893    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj" event={"ID":"293cbeeb-1f98-40f1-8776-f613368c1b92","Type":"ContainerStarted","Data":"edbbb2f567f8f561ca4fd7bb5374a7a0656c183599c5eded7e35dbe2f4306dfc"}
Sep 08 16:24:31.577821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:31.577754    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zdzzj" podStartSLOduration=1.972571502 podStartE2EDuration="6.577740032s" totalImagesPullingTime="4.60516853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:31.576048111 +0000 UTC m=+701.220124803" watchObservedRunningTime="2026-09-08 16:24:31.577740032 +0000 UTC m=+701.221816725"
Sep 08 16:24:33.221609 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:33.221570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:33.320228 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:33.320194    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:35.430707 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.430670    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"]
Sep 08 16:24:35.435189 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.435168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.437928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.437910    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sm84v\""
Sep 08 16:24:35.449145 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.449121    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"]
Sep 08 16:24:35.548084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.548051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.548339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.548097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmg6\" (UniqueName: \"kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.649102 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.649065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.649271 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.649115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmg6\" (UniqueName: \"kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.649466 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.649445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.663492 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.663453    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmg6\" (UniqueName: \"kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6\") pod \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:35.745043 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:35.744966    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:36.022900 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.022847    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"]
Sep 08 16:24:36.025932 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:24:36.025904    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3717f72a_8d63_4293_8b7d_d8cf6be958e8.slice/crio-d5ebc4c66c248841c1bf0c81a93750af36e9f0ce4ec26bd00ab090e6aa1fcc3c WatchSource:0}: Error finding container d5ebc4c66c248841c1bf0c81a93750af36e9f0ce4ec26bd00ab090e6aa1fcc3c: Status 404 returned error can't find the container with id d5ebc4c66c248841c1bf0c81a93750af36e9f0ce4ec26bd00ab090e6aa1fcc3c
Sep 08 16:24:36.173893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.173853    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"]
Sep 08 16:24:36.179421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.179389    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"]
Sep 08 16:24:36.207406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.207380    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:24:36.211391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.211373    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.223347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.223318    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"]
Sep 08 16:24:36.223592 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.223569    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" containerName="manager" containerID="cri-o://54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000" gracePeriod=2
Sep 08 16:24:36.236882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.236794    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"]
Sep 08 16:24:36.241416 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.241057    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:24:36.266819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.266762    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"]
Sep 08 16:24:36.267281 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.267261    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" containerName="manager"
Sep 08 16:24:36.267379 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.267285    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" containerName="manager"
Sep 08 16:24:36.267438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.267386    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" containerName="manager"
Sep 08 16:24:36.270646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.270626    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:36.271305 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.271280    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-n8bd7\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.274062 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.274006    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-n8bd7\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.291358 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.291334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"]
Sep 08 16:24:36.356002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.355969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.356174 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.356010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwfv\" (UniqueName: \"kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.356174 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.356037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5zg\" (UniqueName: \"kubernetes.io/projected/5c1a059d-b5fa-4138-bff8-deb81ce7c068-kube-api-access-jj5zg\") pod \"limitador-operator-controller-manager-74495c69f8-822pj\" (UID: \"5c1a059d-b5fa-4138-bff8-deb81ce7c068\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:36.456479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.456451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.456479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.456494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwfv\" (UniqueName: \"kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.456999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.456528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5zg\" (UniqueName: \"kubernetes.io/projected/5c1a059d-b5fa-4138-bff8-deb81ce7c068-kube-api-access-jj5zg\") pod \"limitador-operator-controller-manager-74495c69f8-822pj\" (UID: \"5c1a059d-b5fa-4138-bff8-deb81ce7c068\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:36.456999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.456918    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.458697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.458677    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.460982 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.460943    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-n8bd7\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.470020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.469997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5zg\" (UniqueName: \"kubernetes.io/projected/5c1a059d-b5fa-4138-bff8-deb81ce7c068-kube-api-access-jj5zg\") pod \"limitador-operator-controller-manager-74495c69f8-822pj\" (UID: \"5c1a059d-b5fa-4138-bff8-deb81ce7c068\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:36.470140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.470035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwfv\" (UniqueName: \"kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv\") pod \"kuadrant-operator-controller-manager-5f685c496f-tgvcd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.521523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.521485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:36.557655 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.557616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpvvz\" (UniqueName: \"kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz\") pod \"9140e2ba-ec45-471f-9f83-4a9dec4cca44\" (UID: \"9140e2ba-ec45-471f-9f83-4a9dec4cca44\") "
Sep 08 16:24:36.560190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.560151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz" (OuterVolumeSpecName: "kube-api-access-tpvvz") pod "9140e2ba-ec45-471f-9f83-4a9dec4cca44" (UID: "9140e2ba-ec45-471f-9f83-4a9dec4cca44"). InnerVolumeSpecName "kube-api-access-tpvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:24:36.586559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.586333    2571 generic.go:356] "Generic (PLEG): container finished" podID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" containerID="54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000" exitCode=0
Sep 08 16:24:36.586559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.586408    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.586559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.586441    2571 scope.go:117] "RemoveContainer" containerID="54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000"
Sep 08 16:24:36.589481 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.589448    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-n8bd7\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.598410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.598384    2571 scope.go:117] "RemoveContainer" containerID="54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000"
Sep 08 16:24:36.598738 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.598706    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000\": container with ID starting with 54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000 not found: ID does not exist" containerID="54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000"
Sep 08 16:24:36.598856 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.598749    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000"} err="failed to get container status \"54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000\": rpc error: code = NotFound desc = could not find container \"54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000\": container with ID starting with 54a71290cdc9d55a4a0458baf12b310f5a299f854524582ebe89075dae695000 not found: ID does not exist"
Sep 08 16:24:36.600644 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:36.600619    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-n8bd7\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n8bd7"
Sep 08 16:24:36.606028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.606008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:36.660008 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.659353    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvvz\" (UniqueName: \"kubernetes.io/projected/9140e2ba-ec45-471f-9f83-4a9dec4cca44-kube-api-access-tpvvz\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:24:36.813681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.813543    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:24:36.818357 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:24:36.818328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3e9bb9_22bf_4fc5_b5ef_653a4062d7bd.slice/crio-9b03238b27b83fdb85c830ae5aaf83817c22eb002d160d88ba60af3cd4df179a WatchSource:0}: Error finding container 9b03238b27b83fdb85c830ae5aaf83817c22eb002d160d88ba60af3cd4df179a: Status 404 returned error can't find the container with id 9b03238b27b83fdb85c830ae5aaf83817c22eb002d160d88ba60af3cd4df179a
Sep 08 16:24:36.889016 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.888989    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9140e2ba-ec45-471f-9f83-4a9dec4cca44" path="/var/lib/kubelet/pods/9140e2ba-ec45-471f-9f83-4a9dec4cca44/volumes"
Sep 08 16:24:36.908235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:36.908208    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"]
Sep 08 16:24:36.911171 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:24:36.911143    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1a059d_b5fa_4138_bff8_deb81ce7c068.slice/crio-f497d048aed254cb1f49641db0de4ab4a415823a933b52cb58c87aa2446e5dbc WatchSource:0}: Error finding container f497d048aed254cb1f49641db0de4ab4a415823a933b52cb58c87aa2446e5dbc: Status 404 returned error can't find the container with id f497d048aed254cb1f49641db0de4ab4a415823a933b52cb58c87aa2446e5dbc
Sep 08 16:24:37.595887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:37.595843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj" event={"ID":"5c1a059d-b5fa-4138-bff8-deb81ce7c068","Type":"ContainerStarted","Data":"a58c20ea581038429e27b482bc4837893df4b2eed94375c6824aec90043bf6ce"}
Sep 08 16:24:37.596490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:37.595905    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj" event={"ID":"5c1a059d-b5fa-4138-bff8-deb81ce7c068","Type":"ContainerStarted","Data":"f497d048aed254cb1f49641db0de4ab4a415823a933b52cb58c87aa2446e5dbc"}
Sep 08 16:24:37.596490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:37.595976    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:37.597571 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:37.597539    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" event={"ID":"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd","Type":"ContainerStarted","Data":"9b03238b27b83fdb85c830ae5aaf83817c22eb002d160d88ba60af3cd4df179a"}
Sep 08 16:24:37.626053 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:37.625981    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj" podStartSLOduration=1.62596173 podStartE2EDuration="1.62596173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:37.62337576 +0000 UTC m=+707.267452452" watchObservedRunningTime="2026-09-08 16:24:37.62596173 +0000 UTC m=+707.270038425"
Sep 08 16:24:39.969654 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:39.969624    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:40.617556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.617497    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" containerName="manager" containerID="cri-o://afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db" gracePeriod=2
Sep 08 16:24:40.619578 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.619549    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" event={"ID":"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd","Type":"ContainerStarted","Data":"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"}
Sep 08 16:24:40.619705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.619683    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:40.620100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.620079    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:40.647410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.647354    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" podStartSLOduration=1.72376385 podStartE2EDuration="4.647334248s" totalImagesPullingTime="2.923570398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:40.645293614 +0000 UTC m=+710.289370305" watchObservedRunningTime="2026-09-08 16:24:40.647334248 +0000 UTC m=+710.291410941"
Sep 08 16:24:40.851230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:40.851206    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:40.858122 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:40.858098    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:40.887409 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:40.887333    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:41.000634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.000606    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmg6\" (UniqueName: \"kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6\") pod \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") "
Sep 08 16:24:41.001022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.000690    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume\") pod \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\" (UID: \"3717f72a-8d63-4293-8b7d-d8cf6be958e8\") "
Sep 08 16:24:41.001022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.000961    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3717f72a-8d63-4293-8b7d-d8cf6be958e8" (UID: "3717f72a-8d63-4293-8b7d-d8cf6be958e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:24:41.002923 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.002899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6" (OuterVolumeSpecName: "kube-api-access-rtmg6") pod "3717f72a-8d63-4293-8b7d-d8cf6be958e8" (UID: "3717f72a-8d63-4293-8b7d-d8cf6be958e8"). InnerVolumeSpecName "kube-api-access-rtmg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:24:41.101515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.101479    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3717f72a-8d63-4293-8b7d-d8cf6be958e8-extensions-socket-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:24:41.101515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.101507    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmg6\" (UniqueName: \"kubernetes.io/projected/3717f72a-8d63-4293-8b7d-d8cf6be958e8-kube-api-access-rtmg6\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:24:41.625490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.625411    2571 generic.go:356] "Generic (PLEG): container finished" podID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" containerID="afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db" exitCode=2
Sep 08 16:24:41.625490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.625454    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:41.625490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.625472    2571 scope.go:117] "RemoveContainer" containerID="afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db"
Sep 08 16:24:41.633977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.633962    2571 scope.go:117] "RemoveContainer" containerID="afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db"
Sep 08 16:24:41.634233 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:41.634215    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db\": container with ID starting with afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db not found: ID does not exist" containerID="afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db"
Sep 08 16:24:41.634293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:41.634242    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db"} err="failed to get container status \"afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db\": rpc error: code = NotFound desc = could not find container \"afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db\": container with ID starting with afc8a77208aa353b22c8683d5074e94a7fe5d4c6d231b7b3b20ca72d3b1ca3db not found: ID does not exist"
Sep 08 16:24:41.636353 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:24:41.636330    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-jbxnl\" is forbidden: User \"system:node:ip-10-0-135-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-163.ec2.internal' and this object" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jbxnl"
Sep 08 16:24:42.887778 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:42.887743    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" path="/var/lib/kubelet/pods/3717f72a-8d63-4293-8b7d-d8cf6be958e8/volumes"
Sep 08 16:24:43.671967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:43.671933    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:43.768719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:43.768672    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:43.869428 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:43.869371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:43.935248 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:43.935160    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:45.266620 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:45.266589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:47.677371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:47.677330    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:48.604632 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:48.604606    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-822pj"
Sep 08 16:24:48.887707 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:48.887619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:48.990539 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:48.990508    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:51.627745 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:51.627713    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:24:53.861513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:53.861474    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:53.963360 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:53.963324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:54.249308 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:54.249216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:54.355214 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:54.355170    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:54.674711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:54.674677    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:24:54.781808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:24:54.781752    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:04.338015 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.337978    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:25:04.338417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.338178    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" podUID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" containerName="manager" containerID="cri-o://a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c" gracePeriod=10
Sep 08 16:25:04.581322 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.581293    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:25:04.694309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.694215    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume\") pod \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") "
Sep 08 16:25:04.694487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.694317    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkwfv\" (UniqueName: \"kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv\") pod \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\" (UID: \"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd\") "
Sep 08 16:25:04.694608 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.694582    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" (UID: "bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:25:04.696638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.696611    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv" (OuterVolumeSpecName: "kube-api-access-fkwfv") pod "bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" (UID: "bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd"). InnerVolumeSpecName "kube-api-access-fkwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:25:04.733720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.733689    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" containerID="a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c" exitCode=0
Sep 08 16:25:04.733871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.733745    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"
Sep 08 16:25:04.733871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.733793    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" event={"ID":"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd","Type":"ContainerDied","Data":"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"}
Sep 08 16:25:04.733871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.733848    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd" event={"ID":"bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd","Type":"ContainerDied","Data":"9b03238b27b83fdb85c830ae5aaf83817c22eb002d160d88ba60af3cd4df179a"}
Sep 08 16:25:04.733871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.733870    2571 scope.go:117] "RemoveContainer" containerID="a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"
Sep 08 16:25:04.742881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.742862    2571 scope.go:117] "RemoveContainer" containerID="a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"
Sep 08 16:25:04.743134 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:25:04.743116    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c\": container with ID starting with a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c not found: ID does not exist" containerID="a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"
Sep 08 16:25:04.743182 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.743143    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c"} err="failed to get container status \"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c\": rpc error: code = NotFound desc = could not find container \"a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c\": container with ID starting with a79f27f2601342cadf3d10e7e617a202542822ea51a9db477592e81353f50d1c not found: ID does not exist"
Sep 08 16:25:04.759006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.758978    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:25:04.763423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.763405    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-tgvcd"]
Sep 08 16:25:04.794977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.794953    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwfv\" (UniqueName: \"kubernetes.io/projected/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-kube-api-access-fkwfv\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:25:04.794977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.794974    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd-extensions-socket-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:25:04.887913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:04.887882    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" path="/var/lib/kubelet/pods/bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd/volumes"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.986865    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"]
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987591    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" containerName="manager"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987608    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" containerName="manager"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987621    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" containerName="manager"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987630    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" containerName="manager"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987820    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3717f72a-8d63-4293-8b7d-d8cf6be958e8" containerName="manager"
Sep 08 16:25:07.990341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.987833    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3e9bb9-22bf-4fc5-b5ef-653a4062d7bd" containerName="manager"
Sep 08 16:25:07.995146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.995123    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:07.998147 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.998112    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 16:25:07.998274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.998114    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 16:25:07.998593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.998573    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 16:25:07.998690 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:07.998579    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ttlz8\""
Sep 08 16:25:08.003603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.003581    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"]
Sep 08 16:25:08.028398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.028373    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:08.122238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122238    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122330    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc6n\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-kube-api-access-hdc6n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.122628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.122521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223790 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223743    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223790 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223846    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc6n\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-kube-api-access-hdc6n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.223955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.223968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224406    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224468    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.224534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.224483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.226514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.226493    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.226607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.226528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.232220 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.232188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.233151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.233128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc6n\" (UniqueName: \"kubernetes.io/projected/02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38-kube-api-access-hdc6n\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hhl7g\" (UID: \"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.306601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.306522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:08.372942 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.372905    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:08.582425 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.582399    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"]
Sep 08 16:25:08.584227 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:25:08.584202    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02036e9c_f04b_4f6d_aa6e_7e45d1bfbf38.slice/crio-d08138e6767832d728dd446d61d549c9f9456efae185f5ca6f8c6a027d10ffdf WatchSource:0}: Error finding container d08138e6767832d728dd446d61d549c9f9456efae185f5ca6f8c6a027d10ffdf: Status 404 returned error can't find the container with id d08138e6767832d728dd446d61d549c9f9456efae185f5ca6f8c6a027d10ffdf
Sep 08 16:25:08.752642 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:08.752601    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" event={"ID":"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38","Type":"ContainerStarted","Data":"d08138e6767832d728dd446d61d549c9f9456efae185f5ca6f8c6a027d10ffdf"}
Sep 08 16:25:10.394283 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:10.394246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:10.461663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:10.461632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:11.467507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:11.467475    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:25:11.467752 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:11.467557    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:25:11.467752 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:11.467588    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:25:11.771157 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:11.771072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" event={"ID":"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38","Type":"ContainerStarted","Data":"c1f841b235cb91a1c061818b733997b971ab0d606014a07c9f167e854b04ed9b"}
Sep 08 16:25:11.794727 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:11.794682    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podStartSLOduration=1.913610656 podStartE2EDuration="4.794666381s" totalImagesPullingTime="2.881055725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:11.794193905 +0000 UTC m=+741.438270601" watchObservedRunningTime="2026-09-08 16:25:11.794666381 +0000 UTC m=+741.438743072"
Sep 08 16:25:12.306858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:12.306818    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:12.311609 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:12.311583    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:12.775633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:12.775591    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:12.776624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:12.776607    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:25:20.409498 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:20.409441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:20.478579 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:20.478541    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:21.405059 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.405019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:21.477220 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.477185    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:21.576859 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.576823    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:21.680940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.680868    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:25:21.683284 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.683263    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.685805 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.685763    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 16:25:21.696516 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.696491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:25:21.724448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.724424    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:25:21.836744 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.836715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ac54591-ffea-4a02-99fa-939739786af2-config-file\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.836922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.836750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/9ac54591-ffea-4a02-99fa-939739786af2-kube-api-access-nj678\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.937454 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.937384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ac54591-ffea-4a02-99fa-939739786af2-config-file\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.937454 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.937417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/9ac54591-ffea-4a02-99fa-939739786af2-kube-api-access-nj678\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.938079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.938061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ac54591-ffea-4a02-99fa-939739786af2-config-file\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.946141 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.946114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/9ac54591-ffea-4a02-99fa-939739786af2-kube-api-access-nj678\") pod \"limitador-limitador-84994f7ffc-xsdgb\" (UID: \"9ac54591-ffea-4a02-99fa-939739786af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:21.994054 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:21.994031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:22.206564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:22.206477    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:22.264633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:22.264610    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:25:22.267428 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:25:22.267392    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac54591_ffea_4a02_99fa_939739786af2.slice/crio-3f0e438006e827299c4637454128b6a8f18d4ca6305ddfdec3edd09a8173ab5c WatchSource:0}: Error finding container 3f0e438006e827299c4637454128b6a8f18d4ca6305ddfdec3edd09a8173ab5c: Status 404 returned error can't find the container with id 3f0e438006e827299c4637454128b6a8f18d4ca6305ddfdec3edd09a8173ab5c
Sep 08 16:25:22.344683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:22.344652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:22.820451 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:22.820416    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb" event={"ID":"9ac54591-ffea-4a02-99fa-939739786af2","Type":"ContainerStarted","Data":"3f0e438006e827299c4637454128b6a8f18d4ca6305ddfdec3edd09a8173ab5c"}
Sep 08 16:25:23.826337 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:23.826307    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb" event={"ID":"9ac54591-ffea-4a02-99fa-939739786af2","Type":"ContainerStarted","Data":"2010e92f465f9ea53aeaee4c167a918c2bba12c789951e3118ea7e71671b5228"}
Sep 08 16:25:23.826730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:23.826455    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:23.843669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:23.843618    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb" podStartSLOduration=1.496723107 podStartE2EDuration="2.843603726s" totalImagesPullingTime="1.346880619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:23.842501149 +0000 UTC m=+753.486577847" watchObservedRunningTime="2026-09-08 16:25:23.843603726 +0000 UTC m=+753.487680482"
Sep 08 16:25:34.832090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:34.832064    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"
Sep 08 16:25:39.084434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:39.084389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:39.208211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:39.208173    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:39.556384 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:39.556270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:51.868969 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:51.868933    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:52.087304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.087275    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:25:52.391569 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.391538    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:25:52.399820 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.399794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.402728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.402709    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 16:25:52.403854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.403839    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zwblx\""
Sep 08 16:25:52.407577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.407555    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:25:52.483733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.483706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.483871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.483740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjpz\" (UniqueName: \"kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.584310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.584284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.584449 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.584318    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjpz\" (UniqueName: \"kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.586721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.586702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.597221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.597199    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjpz\" (UniqueName: \"kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz\") pod \"authorino-69f7c5df54-4hdpt\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") " pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.709980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.709892    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:25:52.981163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:52.981089    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:25:52.983997 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:25:52.983972    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c7e2bf_ce47_47eb_b3d4_ac783a8910c2.slice/crio-3b5079110542a10b8c4fb5e08c4498fc55cd9e7c44ebe97308eb39f1895bc7ea WatchSource:0}: Error finding container 3b5079110542a10b8c4fb5e08c4498fc55cd9e7c44ebe97308eb39f1895bc7ea: Status 404 returned error can't find the container with id 3b5079110542a10b8c4fb5e08c4498fc55cd9e7c44ebe97308eb39f1895bc7ea
Sep 08 16:25:53.951030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:53.950998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" event={"ID":"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2","Type":"ContainerStarted","Data":"3b5079110542a10b8c4fb5e08c4498fc55cd9e7c44ebe97308eb39f1895bc7ea"}
Sep 08 16:25:55.960314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:55.960279    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" event={"ID":"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2","Type":"ContainerStarted","Data":"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"}
Sep 08 16:25:55.975327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:25:55.975274    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" podStartSLOduration=1.564146848 podStartE2EDuration="3.975256731s" totalImagesPullingTime="2.411109883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:55.975036504 +0000 UTC m=+785.619113198" watchObservedRunningTime="2026-09-08 16:25:55.975256731 +0000 UTC m=+785.619333424"
Sep 08 16:26:00.827267 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:00.827237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:00.924949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:00.924916    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:00.990373 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:00.990335    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:01.110453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:01.110341    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:01.288063 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:01.288027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:03.720060 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:03.720017    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:03.835480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:03.835443    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:35.596893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:35.596798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:35.760937 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:35.760896    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:37.088556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:37.088520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:37.280623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:37.280563    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:37.595467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:37.595427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:38.026031 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:38.025914    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:38.229813 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:38.229761    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:38.535481 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:38.535434    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:38.739385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:38.739347    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:26:47.779808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:47.779746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:48.196541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.196506    2571 generic.go:356] "Generic (PLEG): container finished" podID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerID="c1f841b235cb91a1c061818b733997b971ab0d606014a07c9f167e854b04ed9b" exitCode=0
Sep 08 16:26:48.196708 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.196580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" event={"ID":"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38","Type":"ContainerDied","Data":"c1f841b235cb91a1c061818b733997b971ab0d606014a07c9f167e854b04ed9b"}
Sep 08 16:26:48.196996 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.196982    2571 scope.go:117] "RemoveContainer" containerID="c1f841b235cb91a1c061818b733997b971ab0d606014a07c9f167e854b04ed9b"
Sep 08 16:26:48.198010 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.197978    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:26:48.198129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.198108    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:26:48.198196 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:48.198168    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:26:49.208812 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:49.208758    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" event={"ID":"02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38","Type":"ContainerStarted","Data":"93ffb13e42b1ce5a74280e8eb22b510ac2a0b306e142ae76f49434fcca58a26e"}
Sep 08 16:26:49.307376 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:49.307343    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:26:49.307565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:49.307545    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hhl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 08 16:26:49.307641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:49.307619    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podUID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 08 16:26:50.307250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:50.307215    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hhl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 08 16:26:50.307709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:50.307287    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podUID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 08 16:26:51.307148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:51.307113    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hhl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 08 16:26:51.307311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:51.307181    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podUID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 08 16:26:51.420456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:51.420401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:52.307093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:52.307054    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hhl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 08 16:26:52.307265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:52.307125    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podUID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 08 16:26:53.307479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:53.307440    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hhl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 08 16:26:53.307878 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:53.307512    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g" podUID="02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 08 16:26:53.892401 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:53.892360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:26:54.311676 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:54.311600    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:26:54.312153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:54.312009    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:26:54.312790 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:54.312747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hhl7g"
Sep 08 16:26:54.356494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:26:54.356458    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:03.721403 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:03.721369    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:06.732392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:06.732360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:07.317465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:07.317432    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:18.236468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.236433    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"]
Sep 08 16:27:18.239884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.239866    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.242782 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.242741    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 16:27:18.243107 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.243091    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 16:27:18.249725 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.249708    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4pf\""
Sep 08 16:27:18.255442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.255425    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 08 16:27:18.281540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.281513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"]
Sep 08 16:27:18.328980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.328945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.329178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.329024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f32a388-f617-43d4-9e32-66bbd6e16073-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.329178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.329057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.329178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.329079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ttn\" (UniqueName: \"kubernetes.io/projected/2f32a388-f617-43d4-9e32-66bbd6e16073-kube-api-access-x9ttn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.329178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.329113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.329178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.329143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.365875 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.365838    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:18.430515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.430515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.430754 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430568    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.430754 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f32a388-f617-43d4-9e32-66bbd6e16073-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.430754 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.430754 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.430641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ttn\" (UniqueName: \"kubernetes.io/projected/2f32a388-f617-43d4-9e32-66bbd6e16073-kube-api-access-x9ttn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.431092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.431062    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.431203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.431091    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.431203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.431190    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.433095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.433071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f32a388-f617-43d4-9e32-66bbd6e16073-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.433188 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.433173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f32a388-f617-43d4-9e32-66bbd6e16073-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.441988 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.441969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ttn\" (UniqueName: \"kubernetes.io/projected/2f32a388-f617-43d4-9e32-66bbd6e16073-kube-api-access-x9ttn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5\" (UID: \"2f32a388-f617-43d4-9e32-66bbd6e16073\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.549866 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.549742    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:18.868122 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.868091    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"]
Sep 08 16:27:18.871238 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:27:18.871202    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f32a388_f617_43d4_9e32_66bbd6e16073.slice/crio-724c82e38be23bf47638e9e34c76719af73004e984750eba1e5215a34b243bf1 WatchSource:0}: Error finding container 724c82e38be23bf47638e9e34c76719af73004e984750eba1e5215a34b243bf1: Status 404 returned error can't find the container with id 724c82e38be23bf47638e9e34c76719af73004e984750eba1e5215a34b243bf1
Sep 08 16:27:18.873171 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:18.873150    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:27:19.296827 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:19.296698    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:19.344958 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:19.344904    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5" event={"ID":"2f32a388-f617-43d4-9e32-66bbd6e16073","Type":"ContainerStarted","Data":"724c82e38be23bf47638e9e34c76719af73004e984750eba1e5215a34b243bf1"}
Sep 08 16:27:20.350977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:20.350939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5" event={"ID":"2f32a388-f617-43d4-9e32-66bbd6e16073","Type":"ContainerStarted","Data":"52b2552d3c8e9aff6e2ac9760042889f3529205c7032e56973fef213d404af68"}
Sep 08 16:27:22.270615 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:22.270581    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:23.371846 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:23.371788    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5" podStartSLOduration=4.225195798 podStartE2EDuration="5.371746076s" totalImagesPullingTime="1.146550278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:20.544034158 +0000 UTC m=+870.188110850" watchObservedRunningTime="2026-09-08 16:27:23.371746076 +0000 UTC m=+873.015822768"
Sep 08 16:27:23.374647 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:23.374605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:28.550025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:28.549986    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:28.550442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:28.550043    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:28.562718 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:28.562690    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:29.402613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:29.402583    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5"
Sep 08 16:27:29.992025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:29.991989    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:30.860261 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:30.860226    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:38.437030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:38.436992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:40.201355 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:40.201319    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:46.747994 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.747911    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"]
Sep 08 16:27:46.751270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.751250    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.758162 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.758135    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 16:27:46.781858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.781816    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"]
Sep 08 16:27:46.891093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44425b-f61d-494e-b74c-7eda55ab5153-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.891093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmd5f\" (UniqueName: \"kubernetes.io/projected/2a44425b-f61d-494e-b74c-7eda55ab5153-kube-api-access-kmd5f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.891293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.891293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.891293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.891390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.891296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.905660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.905631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:27:46.992467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44425b-f61d-494e-b74c-7eda55ab5153-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.992467 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmd5f\" (UniqueName: \"kubernetes.io/projected/2a44425b-f61d-494e-b74c-7eda55ab5153-kube-api-access-kmd5f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.992721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.992721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992521    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.992721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.992721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.992570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.993037 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.993014    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.993113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.993086    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.993113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.993094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.995063 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.995042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44425b-f61d-494e-b74c-7eda55ab5153-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:46.995269 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:46.995253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44425b-f61d-494e-b74c-7eda55ab5153-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:47.011899 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:47.011837    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmd5f\" (UniqueName: \"kubernetes.io/projected/2a44425b-f61d-494e-b74c-7eda55ab5153-kube-api-access-kmd5f\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb\" (UID: \"2a44425b-f61d-494e-b74c-7eda55ab5153\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:47.060569 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:47.060522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:47.368586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:47.368553    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"]
Sep 08 16:27:47.372058 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:27:47.372030    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a44425b_f61d_494e_b74c_7eda55ab5153.slice/crio-408dcbb5ff7986e65f197c2e7cd458dda2cdf7f960f10020c7c82b3c0115926f WatchSource:0}: Error finding container 408dcbb5ff7986e65f197c2e7cd458dda2cdf7f960f10020c7c82b3c0115926f: Status 404 returned error can't find the container with id 408dcbb5ff7986e65f197c2e7cd458dda2cdf7f960f10020c7c82b3c0115926f
Sep 08 16:27:47.475452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:47.475412    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb" event={"ID":"2a44425b-f61d-494e-b74c-7eda55ab5153","Type":"ContainerStarted","Data":"408dcbb5ff7986e65f197c2e7cd458dda2cdf7f960f10020c7c82b3c0115926f"}
Sep 08 16:27:48.481553 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:48.481518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb" event={"ID":"2a44425b-f61d-494e-b74c-7eda55ab5153","Type":"ContainerStarted","Data":"58bf224167638f6ff60456cf7e06e9ff5a479365d0119422b0339e11cee346c4"}
Sep 08 16:27:48.518033 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:48.517979    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb" podStartSLOduration=2.329159947 podStartE2EDuration="2.517961873s" totalImagesPullingTime="188.801926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:48.517627217 +0000 UTC m=+898.161703911" watchObservedRunningTime="2026-09-08 16:27:48.517961873 +0000 UTC m=+898.162038565"
Sep 08 16:27:49.008381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:49.008344    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:27:50.841376 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:50.841346    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:27:50.843119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:50.843096    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:27:57.061208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:57.061171    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:57.061650 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:57.061216    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:57.073558 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:57.073533    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:27:57.534699 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:27:57.534662    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb"
Sep 08 16:28:03.859490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:28:03.859450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:28:14.933809 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:28:14.933751    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:28:51.396327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:28:51.396276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:07.567852 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.567818    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:29:07.568484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.567990    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" podUID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" containerName="authorino" containerID="cri-o://8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd" gracePeriod=30
Sep 08 16:29:07.813943 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.813919    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:29:07.834415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.834358    2571 generic.go:356] "Generic (PLEG): container finished" podID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" containerID="8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd" exitCode=0
Sep 08 16:29:07.834415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.834394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" event={"ID":"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2","Type":"ContainerDied","Data":"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"}
Sep 08 16:29:07.834415 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.834404    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69f7c5df54-4hdpt"
Sep 08 16:29:07.834601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.834420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69f7c5df54-4hdpt" event={"ID":"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2","Type":"ContainerDied","Data":"3b5079110542a10b8c4fb5e08c4498fc55cd9e7c44ebe97308eb39f1895bc7ea"}
Sep 08 16:29:07.834601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.834436    2571 scope.go:117] "RemoveContainer" containerID="8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"
Sep 08 16:29:07.842438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.842419    2571 scope.go:117] "RemoveContainer" containerID="8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"
Sep 08 16:29:07.842712 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:29:07.842693    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd\": container with ID starting with 8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd not found: ID does not exist" containerID="8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"
Sep 08 16:29:07.842758 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.842723    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd"} err="failed to get container status \"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd\": rpc error: code = NotFound desc = could not find container \"8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd\": container with ID starting with 8593488f989b9019e0437ad9aa66a82d20381f2ede408c184d95b1138cbb55bd not found: ID does not exist"
Sep 08 16:29:07.867151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.867130    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvjpz\" (UniqueName: \"kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz\") pod \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") "
Sep 08 16:29:07.867235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.867189    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert\") pod \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\" (UID: \"f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2\") "
Sep 08 16:29:07.869207 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.869185    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz" (OuterVolumeSpecName: "kube-api-access-fvjpz") pod "f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" (UID: "f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2"). InnerVolumeSpecName "kube-api-access-fvjpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:29:07.877902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.877877    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" (UID: "f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:29:07.968419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.968387    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjpz\" (UniqueName: \"kubernetes.io/projected/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-kube-api-access-fvjpz\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:29:07.968419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:07.968415    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2-tls-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:29:08.226103 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:08.226070    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:29:08.227556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:08.227527    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69f7c5df54-4hdpt"]
Sep 08 16:29:08.338676 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:08.338632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:08.571242 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:08.571206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:08.892888 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:08.892799    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" path="/var/lib/kubelet/pods/f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2/volumes"
Sep 08 16:29:09.498958 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:09.498918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:09.704204 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:09.704167    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:21.184079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:21.183993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:26.011404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.011366    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:26.115830 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.115799    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:26.337555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.337528    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:29:26.337905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.337893    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" containerName="authorino"
Sep 08 16:29:26.337962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.337906    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" containerName="authorino"
Sep 08 16:29:26.337999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.337972    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c7e2bf-ce47-47eb-b3d4-ac783a8910c2" containerName="authorino"
Sep 08 16:29:26.340826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.340807    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.344702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.344686    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0d9aa4-openshift-default-dockercfg-68lwq\""
Sep 08 16:29:26.365704 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.360729    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:29:26.373534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.373503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:26.424419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b677j\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424577    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.424723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.424676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.510678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.510637    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:26.525389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b677j\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525560    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.525994 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.525968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.526056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.526010    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.526056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.526036    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.526157 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.526139    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.526406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.526385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.527909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.527888    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.528201 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.528180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.536185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.536164    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.536694 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.536675    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b677j\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j\") pod \"e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") " pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.650955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.650874    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:26.949330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.949235    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:29:26.953319 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:26.953293    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b75000b_d506_4b79_bf9f_9dc220901f12.slice/crio-642a214f080b0a83ba5730fdcdb8e64326f6cda7099a575bae0e80caad986649 WatchSource:0}: Error finding container 642a214f080b0a83ba5730fdcdb8e64326f6cda7099a575bae0e80caad986649: Status 404 returned error can't find the container with id 642a214f080b0a83ba5730fdcdb8e64326f6cda7099a575bae0e80caad986649
Sep 08 16:29:26.955352 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.955321    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:26.955417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.955385    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:26.955455 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:26.955415    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:27.369681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.369648    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:27.590426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.590395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:27.860962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.860927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:27.920957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.920920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerStarted","Data":"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"}
Sep 08 16:29:27.921124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.920973    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerStarted","Data":"642a214f080b0a83ba5730fdcdb8e64326f6cda7099a575bae0e80caad986649"}
Sep 08 16:29:27.957706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:27.957659    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podStartSLOduration=1.957643224 podStartE2EDuration="1.957643224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:27.957384233 +0000 UTC m=+997.601460925" watchObservedRunningTime="2026-09-08 16:29:27.957643224 +0000 UTC m=+997.601719916"
Sep 08 16:29:28.652128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:28.652091    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:28.657298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:28.657271    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:28.925606 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:28.925519    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:28.926683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:28.926653    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:30.169841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:30.169809    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:30.378647 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:30.378615    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:29:30.507611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:30.507523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:31.386965 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.386931    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:29:31.390725 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.390699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.397890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.397859    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksssl\""
Sep 08 16:29:31.399045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.399013    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:29:31.399168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.399051    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:29:31.399168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.399060    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0d9aa4-serving-cert\""
Sep 08 16:29:31.399168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.399020    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:29:31.414290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.414265    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:29:31.463919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.463880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:31.471028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.471008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.471130 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.471040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfc6z\" (UniqueName: \"kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.471130 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.471090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.471130 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.471111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.572012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.571974    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.572193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.572021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfc6z\" (UniqueName: \"kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.572193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.572087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.572193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.572118    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.572371 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:29:31.572203    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-serving-cert: secret "maas-api-e2e-inf-a-0d9aa4-serving-cert" not found
Sep 08 16:29:31.572371 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:29:31.572308    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls podName:d2eeeca7-61e8-48e4-8833-3afb9fb60a5d nodeName:}" failed. No retries permitted until 2026-09-08 16:29:32.07228535 +0000 UTC m=+1001.716362020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls") pod "maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" (UID: "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d") : secret "maas-api-e2e-inf-a-0d9aa4-serving-cert" not found
Sep 08 16:29:31.572706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.572684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.574683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.574650    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:31.583557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.583533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:31.584985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:31.584961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfc6z\" (UniqueName: \"kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:32.075689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.075659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:32.078110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.078081    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:32.116971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.116941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:32.246257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.246222    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:32.308494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.308466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:32.590030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.590007    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:29:32.592618 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:32.592579    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2eeeca7_61e8_48e4_8833_3afb9fb60a5d.slice/crio-f7e54b703117b24ac962eca48354601513587077e9d5982e6653030a0515bcb0 WatchSource:0}: Error finding container f7e54b703117b24ac962eca48354601513587077e9d5982e6653030a0515bcb0: Status 404 returned error can't find the container with id f7e54b703117b24ac962eca48354601513587077e9d5982e6653030a0515bcb0
Sep 08 16:29:32.627460 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.627402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:32.945137 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:32.945067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" event={"ID":"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d","Type":"ContainerStarted","Data":"f7e54b703117b24ac962eca48354601513587077e9d5982e6653030a0515bcb0"}
Sep 08 16:29:34.679337 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.679308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:34.766980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.766956    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:34.894453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.894422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:34.957363 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.957284    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" event={"ID":"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d","Type":"ContainerStarted","Data":"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"}
Sep 08 16:29:34.957499 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.957397    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:34.977498 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:34.977456    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" podStartSLOduration=2.272759416 podStartE2EDuration="3.977439367s" totalImagesPullingTime="1.704679951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:34.975559731 +0000 UTC m=+1004.619636424" watchObservedRunningTime="2026-09-08 16:29:34.977439367 +0000 UTC m=+1004.621516060"
Sep 08 16:29:35.211903 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.211825    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:29:35.215537 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.215515    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.222068 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.221907    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ec338109-openshift-default-dockercfg-knbbp\""
Sep 08 16:29:35.241245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.236063    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:29:35.256482 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.256458    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:35.305393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305462    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305630    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbr7\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.305760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.305706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.359709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.359675    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:35.407143 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407260 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407175    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407260 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407195    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbr7\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407260 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407428 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407483 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407738 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407720    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.407921 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407901    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.408003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.408003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.407927    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.408115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.408100    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.409973 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.409953    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.409973 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.409964    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.418064 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.418044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.418156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.418051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbr7\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7\") pod \"e2e-ait-ec338109-openshift-default-777677fc68-n4skr\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") " pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.525173 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.525111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:35.666562 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.666528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:35.815458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.815435    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:29:35.817615 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:35.817589    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0619e549_8978_413d_bb80_cca810a66161.slice/crio-1d3e4d5936ab055b4da68644ae3506811a63272882dd4b63a9f4f6d42b8e64ca WatchSource:0}: Error finding container 1d3e4d5936ab055b4da68644ae3506811a63272882dd4b63a9f4f6d42b8e64ca: Status 404 returned error can't find the container with id 1d3e4d5936ab055b4da68644ae3506811a63272882dd4b63a9f4f6d42b8e64ca
Sep 08 16:29:35.819870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.819834    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:35.819962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.819902    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:35.819962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.819929    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:35.963595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.963562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerStarted","Data":"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"}
Sep 08 16:29:35.963783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.963621    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerStarted","Data":"1d3e4d5936ab055b4da68644ae3506811a63272882dd4b63a9f4f6d42b8e64ca"}
Sep 08 16:29:35.992484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:35.992445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:36.008706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.008663    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podStartSLOduration=1.008643977 podStartE2EDuration="1.008643977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:36.006437826 +0000 UTC m=+1005.650514518" watchObservedRunningTime="2026-09-08 16:29:36.008643977 +0000 UTC m=+1005.652720669"
Sep 08 16:29:36.138175 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.138143    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:36.526198 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.526124    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:36.531682 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.531655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:36.967896 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.967868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:36.968752 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:36.968729    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:37.856622 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:37.856583    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:38.040163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:38.040128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:38.928981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:38.928947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:39.125681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:39.125625    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:39.282576 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:39.282485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:40.140367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:40.140327    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:40.268308 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:40.268268    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:40.456407 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:40.456331    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:29:40.886481 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:40.886343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:40.982065 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:40.982034    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:29:41.084566 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.084533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:41.216028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.215913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:41.610508 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.610472    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:29:41.615019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.614997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.619096 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.619073    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-ec6a8395-kserve-self-signed-certs\""
Sep 08 16:29:41.646469 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.646447    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:29:41.687431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.687595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnmx\" (UniqueName: \"kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.687595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.687595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687567    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.687736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.687736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.687642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.768418 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.768384    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:41.789172 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnmx\" (UniqueName: \"kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.789892 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.789868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.790340 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.790312    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.790469 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.790330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.792339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.792313    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.792928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.792906    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.825325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.825304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnmx\" (UniqueName: \"kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx\") pod \"e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") " pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:41.909714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.909618    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:41.927437 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:41.927412    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:42.024039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.024012    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:29:42.029294 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.029266    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.035030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.032698    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-293166\""
Sep 08 16:29:42.035030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.032860    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-293166-dockercfg-2jkql\""
Sep 08 16:29:42.047177 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.047155    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:29:42.093410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.093375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.093570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.093455    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzm2l\" (UniqueName: \"kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.194423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.194339    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.194556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.194432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzm2l\" (UniqueName: \"kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.195226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.195205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.223746 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.223715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:42.224971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.224949    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzm2l\" (UniqueName: \"kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l\") pod \"payload-processing-e2e-autores-293166-545989b9dc-4xgrn\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") " pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.252311 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.252289    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:29:42.252933 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:42.252914    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c7cf28_a46d_4d6a_9419_7b1ee0b414ef.slice/crio-f04900b14f6cf005e7aa225a55d96d5c9c6bd51082b0e900049894fdbd006e4c WatchSource:0}: Error finding container f04900b14f6cf005e7aa225a55d96d5c9c6bd51082b0e900049894fdbd006e4c: Status 404 returned error can't find the container with id f04900b14f6cf005e7aa225a55d96d5c9c6bd51082b0e900049894fdbd006e4c
Sep 08 16:29:42.345797 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.345747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:42.385794 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.379677    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:42.646530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:42.646381    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:29:42.648570 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:42.648538    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb62fdf_d711_41bc_a139_851d42772008.slice/crio-88750e495b0f66a1f34f0db4d18dfee769502674fdf77e4ebe7deee6f3a29da7 WatchSource:0}: Error finding container 88750e495b0f66a1f34f0db4d18dfee769502674fdf77e4ebe7deee6f3a29da7: Status 404 returned error can't find the container with id 88750e495b0f66a1f34f0db4d18dfee769502674fdf77e4ebe7deee6f3a29da7
Sep 08 16:29:43.004456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:43.004362    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" event={"ID":"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef","Type":"ContainerStarted","Data":"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"}
Sep 08 16:29:43.004456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:43.004410    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" event={"ID":"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef","Type":"ContainerStarted","Data":"f04900b14f6cf005e7aa225a55d96d5c9c6bd51082b0e900049894fdbd006e4c"}
Sep 08 16:29:43.005676 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:43.005651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" event={"ID":"ecb62fdf-d711-41bc-a139-851d42772008","Type":"ContainerStarted","Data":"88750e495b0f66a1f34f0db4d18dfee769502674fdf77e4ebe7deee6f3a29da7"}
Sep 08 16:29:43.022822 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:43.022779    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" podStartSLOduration=2.022751755 podStartE2EDuration="2.022751755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:43.021365171 +0000 UTC m=+1012.665441864" watchObservedRunningTime="2026-09-08 16:29:43.022751755 +0000 UTC m=+1012.666828489"
Sep 08 16:29:43.328413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:43.328322    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:44.927148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:44.927104    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 16:29:44.927531 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:44.927178    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 16:29:44.960640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:44.960601    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:45.193335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:45.193257    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:45.301497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:45.301472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:46.023528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.023500    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" event={"ID":"ecb62fdf-d711-41bc-a139-851d42772008","Type":"ContainerStarted","Data":"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"}
Sep 08 16:29:46.023959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.023596    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:46.025205 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.025181    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerID="1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1" exitCode=0
Sep 08 16:29:46.025304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.025232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerDied","Data":"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"}
Sep 08 16:29:46.025548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.025534    2571 scope.go:117] "RemoveContainer" containerID="1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"
Sep 08 16:29:46.026465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.026440    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:46.026530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.026506    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:46.026584 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.026539    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:46.042003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.041965    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" podStartSLOduration=1.452920738 podStartE2EDuration="4.041952547s" totalImagesPullingTime="2.589031809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:46.039526727 +0000 UTC m=+1015.683603418" watchObservedRunningTime="2026-09-08 16:29:46.041952547 +0000 UTC m=+1015.686029240"
Sep 08 16:29:46.899024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:46.898990    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:47.032084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.032044    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerStarted","Data":"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"}
Sep 08 16:29:47.090528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.090483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:47.225372 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.225271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:47.651941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.651902    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:47.652105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.652055    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 08 16:29:47.652152 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:47.652103    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 08 16:29:48.651470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:48.651420    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 08 16:29:48.651975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:48.651512    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 08 16:29:49.651964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:49.651925    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 08 16:29:49.652331 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:49.651986    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 08 16:29:50.651546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:50.651506    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 08 16:29:50.651717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:50.651570    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 08 16:29:51.651476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:51.651408    2571 patch_prober.go:28] interesting pod/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Sep 08 16:29:51.651910 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:51.651488    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Sep 08 16:29:51.927804 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:51.927712    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:51.928019 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:51.927996    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:51.940510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:51.940484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:52.094902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:52.094868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:29:52.655066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:52.655030    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:52.655501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:52.655301    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:52.655907 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:52.655889    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:29:52.727914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:52.727881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:53.733706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:53.733668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:54.049813 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:54.049702    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:54.183027 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:54.182981    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:54.855167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:54.855128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:55.166131 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.166044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:55.458119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.458020    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:29:55.480956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.480924    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:29:55.486032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.486008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.488891 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.488869    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b41034-openshift-default-dockercfg-9x6gc\""
Sep 08 16:29:55.498196 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.498174    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:29:55.518365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.518337    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:55.525148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525265 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525387 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525387 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.525734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.525569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdjl\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626470    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdjl\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.626814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627164 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627164 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.626862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.627202    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.627218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.627553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627664 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.627639    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.627818 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.627794    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.629106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.629082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.629349 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.629330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.638100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.638073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.638207 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.638158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdjl\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl\") pod \"e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") " pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:55.733527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.733444    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:55.796711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:55.796673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:56.077914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:56.077884    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:29:56.082151 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:29:56.082125    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd674f56f_ae3c_4b9a_bfae_f0916ba454e5.slice/crio-73e102c54c9e58beae44bb02afe7a676b59c71121a4c371e2fcf61ffd61054a8 WatchSource:0}: Error finding container 73e102c54c9e58beae44bb02afe7a676b59c71121a4c371e2fcf61ffd61054a8: Status 404 returned error can't find the container with id 73e102c54c9e58beae44bb02afe7a676b59c71121a4c371e2fcf61ffd61054a8
Sep 08 16:29:56.084867 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:56.084826    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:56.084934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:56.084901    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:56.084934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:56.084929    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:57.034719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.034648    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:29:57.088808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.088704    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerStarted","Data":"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"}
Sep 08 16:29:57.088808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.088801    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerStarted","Data":"73e102c54c9e58beae44bb02afe7a676b59c71121a4c371e2fcf61ffd61054a8"}
Sep 08 16:29:57.091164 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.091138    2571 generic.go:356] "Generic (PLEG): container finished" podID="0619e549-8978-413d-bb80-cca810a66161" containerID="f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e" exitCode=0
Sep 08 16:29:57.091303 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.091184    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerDied","Data":"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"}
Sep 08 16:29:57.091611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.091590    2571 scope.go:117] "RemoveContainer" containerID="f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"
Sep 08 16:29:57.093115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.093077    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:57.093223 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.093166    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:57.093223 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.093211    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:29:57.117547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.117260    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podStartSLOduration=2.117240094 podStartE2EDuration="2.117240094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:57.111801234 +0000 UTC m=+1026.755877927" watchObservedRunningTime="2026-09-08 16:29:57.117240094 +0000 UTC m=+1026.761316786"
Sep 08 16:29:57.797462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.797430    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:57.802791 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:57.802747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:58.097558 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.097526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerStarted","Data":"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"}
Sep 08 16:29:58.098059 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.097696    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:58.098878 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.098862    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:29:58.526346 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.526258    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:29:58.526500 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.526448    2571 patch_prober.go:28] interesting pod/e2e-ait-ec338109-openshift-default-777677fc68-n4skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 16:29:58.526547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:58.526498    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 16:29:59.018378 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:59.018343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:59.142664 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:59.142626    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:29:59.525961 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:59.525923    2571 patch_prober.go:28] interesting pod/e2e-ait-ec338109-openshift-default-777677fc68-n4skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 16:29:59.526165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:29:59.525988    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 16:30:00.149015 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.148987    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"]
Sep 08 16:30:00.151347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.151320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"
Sep 08 16:30:00.154395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.154377    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:30:00.161160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.161136    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"
Sep 08 16:30:00.169611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.169581    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:30:00.206545 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.206512    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.210966 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.210934    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"]
Sep 08 16:30:00.212592 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.212557    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b41034-serving-cert\""
Sep 08 16:30:00.229277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.229212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:30:00.235620 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.235596    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:00.303881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.303848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ncl\" (UniqueName: \"kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.304090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.304074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.304236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.304212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.304347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.304325    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.325761 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.325652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.405183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.405275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ncl\" (UniqueName: \"kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:30:00.405288    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-serving-cert: secret "maas-api-e2e-inf-b-b41034-serving-cert" not found
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.405301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.405328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.405485 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:30:00.405348    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls podName:606ef2f4-b8c4-4907-9698-67d78cb0456f nodeName:}" failed. No retries permitted until 2026-09-08 16:30:00.905330976 +0000 UTC m=+1030.549407646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls") pod "maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" (UID: "606ef2f4-b8c4-4907-9698-67d78cb0456f") : secret "maas-api-e2e-inf-b-b41034-serving-cert" not found
Sep 08 16:30:00.405930 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.405912    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.407988 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.407963    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.414829 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.414811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ncl\" (UniqueName: \"kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.479345 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.479262    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"]
Sep 08 16:30:00.491563 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:30:00.491536    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1c0e75_f5e4_40dd_97ba_0a75db408df7.slice/crio-1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347 WatchSource:0}: Error finding container 1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347: Status 404 returned error can't find the container with id 1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347
Sep 08 16:30:00.525663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.525637    2571 patch_prober.go:28] interesting pod/e2e-ait-ec338109-openshift-default-777677fc68-n4skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 16:30:00.525753 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.525679    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 16:30:00.684837 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.684725    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:00.878793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.878310    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:00.911099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.911061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:00.913663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:00.913636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") pod \"maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:01.014734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.014623    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:01.113132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.113104    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl" event={"ID":"3a1c0e75-f5e4-40dd-97ba-0a75db408df7","Type":"ContainerStarted","Data":"1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347"}
Sep 08 16:30:01.122260 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.122236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:01.438424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.438401    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:30:01.440271 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:30:01.440237    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606ef2f4_b8c4_4907_9698_67d78cb0456f.slice/crio-545d4efef16071876056e25643e7ac4521415e4f8ac88eca85d4923ca99a1830 WatchSource:0}: Error finding container 545d4efef16071876056e25643e7ac4521415e4f8ac88eca85d4923ca99a1830: Status 404 returned error can't find the container with id 545d4efef16071876056e25643e7ac4521415e4f8ac88eca85d4923ca99a1830
Sep 08 16:30:01.525928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.525898    2571 patch_prober.go:28] interesting pod/e2e-ait-ec338109-openshift-default-777677fc68-n4skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 16:30:01.526076 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.525952    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 16:30:01.718292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:01.718197    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:02.110269 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.110175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:02.120784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.120713    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" event={"ID":"606ef2f4-b8c4-4907-9698-67d78cb0456f","Type":"ContainerStarted","Data":"545d4efef16071876056e25643e7ac4521415e4f8ac88eca85d4923ca99a1830"}
Sep 08 16:30:02.316158 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.316127    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:02.526090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.525924    2571 patch_prober.go:28] interesting pod/e2e-ait-ec338109-openshift-default-777677fc68-n4skr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 16:30:02.526090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.525972    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 16:30:02.687890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:02.687820    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:03.530213 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:03.530177    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:30:03.530536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:03.530451    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:30:03.531268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:03.531249    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:30:03.766830 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:03.766710    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:04.133227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.133185    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" event={"ID":"606ef2f4-b8c4-4907-9698-67d78cb0456f","Type":"ContainerStarted","Data":"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"}
Sep 08 16:30:04.133431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.133299    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:04.135104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.135077    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl" event={"ID":"3a1c0e75-f5e4-40dd-97ba-0a75db408df7","Type":"ContainerStarted","Data":"e32897a84221f25e17e5ae5620a4c0006d3a0969737ca605daec5b4558675768"}
Sep 08 16:30:04.154303 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.154257    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" podStartSLOduration=2.422093195 podStartE2EDuration="4.154242527s" totalImagesPullingTime="1.732149332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:04.152325054 +0000 UTC m=+1033.796401740" watchObservedRunningTime="2026-09-08 16:30:04.154242527 +0000 UTC m=+1033.798319219"
Sep 08 16:30:04.169059 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.169017    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl" podStartSLOduration=1.4497518440000001 podStartE2EDuration="4.169004169s" totalImagesPullingTime="2.719252325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:04.16758904 +0000 UTC m=+1033.811665733" watchObservedRunningTime="2026-09-08 16:30:04.169004169 +0000 UTC m=+1033.813080860"
Sep 08 16:30:04.668931 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:04.668891    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:05.140877 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:05.140839    2571 generic.go:356] "Generic (PLEG): container finished" podID="3a1c0e75-f5e4-40dd-97ba-0a75db408df7" containerID="e32897a84221f25e17e5ae5620a4c0006d3a0969737ca605daec5b4558675768" exitCode=0
Sep 08 16:30:05.141079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:05.140934    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl" event={"ID":"3a1c0e75-f5e4-40dd-97ba-0a75db408df7","Type":"ContainerDied","Data":"e32897a84221f25e17e5ae5620a4c0006d3a0969737ca605daec5b4558675768"}
Sep 08 16:30:05.629409 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:05.629374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:06.273842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:06.273818    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"
Sep 08 16:30:07.151697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:07.151618    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"
Sep 08 16:30:07.151697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:07.151627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl" event={"ID":"3a1c0e75-f5e4-40dd-97ba-0a75db408df7","Type":"ContainerDied","Data":"1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347"}
Sep 08 16:30:07.157733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:07.152082    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1553d6a219d377d1fe9904847b49fde7bf7ba6f1de376343b14700e825b4a347"
Sep 08 16:30:08.125583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:08.125551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:08.879348 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:08.879296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:09.081425 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:09.081393    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:30:10.146604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:10.146577    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:30:19.238690 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.238565    2571 generic.go:356] "Generic (PLEG): container finished" podID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerID="9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a" exitCode=0
Sep 08 16:30:19.238690 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.238638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerDied","Data":"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"}
Sep 08 16:30:19.239286 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.239210    2571 scope.go:117] "RemoveContainer" containerID="9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"
Sep 08 16:30:19.240398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.240366    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:19.240480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.240440    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:19.240517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.240484    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:19.513047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.512949    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:19.592141 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.592104    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:19.653146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.653112    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:19.762360 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.762318    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:19.826510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.826478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:19.941482 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.941451    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:30:19.941702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.941680    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy" containerID="cri-o://4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61" gracePeriod=30
Sep 08 16:30:19.987541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:19.987501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:20.246294 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:20.246210    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerStarted","Data":"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"}
Sep 08 16:30:20.405014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:20.404975    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:20.797201 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:20.797162    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:30:20.797432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:20.797407    2571 patch_prober.go:28] interesting pod/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 08 16:30:20.797515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:20.797464    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 08 16:30:21.337571 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:21.337536    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:21.436156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:21.436124    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:21.536329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:21.536297    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:21.797708 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:21.797612    2571 patch_prober.go:28] interesting pod/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 08 16:30:21.797708 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:21.797672    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 08 16:30:22.797442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:22.797405    2571 patch_prober.go:28] interesting pod/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 08 16:30:22.797852 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:22.797460    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 08 16:30:23.444101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.444065    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:23.545055 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.545018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:23.797872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.797725    2571 patch_prober.go:28] interesting pod/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 08 16:30:23.798324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.797888    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 08 16:30:23.820554 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.820436    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:23.959298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:23.959264    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:24.049145 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:24.049055    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:24.375240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:24.375198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:24.470665 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:24.470601    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:24.798110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:24.798018    2571 patch_prober.go:28] interesting pod/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 08 16:30:24.798110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:24.798093    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 08 16:30:25.235925 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.235903    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:30:25.286117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.284279    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:30:25.294209 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.294176    2571 generic.go:356] "Generic (PLEG): container finished" podID="0619e549-8978-413d-bb80-cca810a66161" containerID="4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61" exitCode=0
Sep 08 16:30:25.294391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.294238    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"
Sep 08 16:30:25.294391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.294248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerDied","Data":"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"}
Sep 08 16:30:25.294391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.294300    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr" event={"ID":"0619e549-8978-413d-bb80-cca810a66161","Type":"ContainerDied","Data":"1d3e4d5936ab055b4da68644ae3506811a63272882dd4b63a9f4f6d42b8e64ca"}
Sep 08 16:30:25.294391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.294324    2571 scope.go:117] "RemoveContainer" containerID="4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"
Sep 08 16:30:25.303980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.303961    2571 scope.go:117] "RemoveContainer" containerID="f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"
Sep 08 16:30:25.312064 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.312045    2571 scope.go:117] "RemoveContainer" containerID="4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"
Sep 08 16:30:25.312319 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:30:25.312301    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61\": container with ID starting with 4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61 not found: ID does not exist" containerID="4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"
Sep 08 16:30:25.312364 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.312329    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61"} err="failed to get container status \"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61\": rpc error: code = NotFound desc = could not find container \"4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61\": container with ID starting with 4ec15a90bacd4e3ad0776bf9e50436479234e09fa5ba95c0b602ae675c1e5f61 not found: ID does not exist"
Sep 08 16:30:25.312364 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.312346    2571 scope.go:117] "RemoveContainer" containerID="f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"
Sep 08 16:30:25.312594 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:30:25.312566    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e\": container with ID starting with f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e not found: ID does not exist" containerID="f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"
Sep 08 16:30:25.312695 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.312600    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e"} err="failed to get container status \"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e\": rpc error: code = NotFound desc = could not find container \"f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e\": container with ID starting with f12604f5a3470abec4b0019e9db2c782e05104a9bf7e5eafe1e6ae45fb5fbd4e not found: ID does not exist"
Sep 08 16:30:25.355023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355003    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355041    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355070    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355105    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbr7\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355275 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355137    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355275 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355167    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355275 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355210    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355275 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355302    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket\") pod \"0619e549-8978-413d-bb80-cca810a66161\" (UID: \"0619e549-8978-413d-bb80-cca810a66161\") "
Sep 08 16:30:25.355476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355310    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data" (OuterVolumeSpecName: "istio-data") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:30:25.355476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355376    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:30:25.355635 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355503    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:30:25.356056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355881    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.356056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355907    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.356056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355922    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.356056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355922    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:30:25.356056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.355971    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:25.357678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.357636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:30:25.357786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.357724    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7" (OuterVolumeSpecName: "kube-api-access-xvbr7") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "kube-api-access-xvbr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:25.357948 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.357924    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token" (OuterVolumeSpecName: "istio-token") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:25.358046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.358022    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0619e549-8978-413d-bb80-cca810a66161" (UID: "0619e549-8978-413d-bb80-cca810a66161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:30:25.457290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457253    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.457290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457287    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbr7\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-kube-api-access-xvbr7\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.457290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457298    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0619e549-8978-413d-bb80-cca810a66161-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.457519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457306    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0619e549-8978-413d-bb80-cca810a66161-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.457519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457315    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0619e549-8978-413d-bb80-cca810a66161-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.457519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.457324    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0619e549-8978-413d-bb80-cca810a66161-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:30:25.621618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.621582    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:30:25.628998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.628970    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ec338109-openshift-default-777677fc68-n4skr"]
Sep 08 16:30:25.800981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.800947    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:30:25.801343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.801192    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:30:25.801869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:25.801850    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:30:26.888234 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:26.888201    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0619e549-8978-413d-bb80-cca810a66161" path="/var/lib/kubelet/pods/0619e549-8978-413d-bb80-cca810a66161/volumes"
Sep 08 16:30:40.287329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:40.287288    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:30:54.697120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:54.697087    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:54.796916 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:54.796878    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:54.911270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:54.911216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:55.047998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.047907    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:55.416839 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.416805    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:55.684385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.684310    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:30:55.684994 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.684973    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1c0e75-f5e4-40dd-97ba-0a75db408df7" containerName="cleanup"
Sep 08 16:30:55.685043 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685000    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1c0e75-f5e4-40dd-97ba-0a75db408df7" containerName="cleanup"
Sep 08 16:30:55.685043 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685025    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.685043 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685035    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.685138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685062    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.685138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685071    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.685202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685174    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1c0e75-f5e4-40dd-97ba-0a75db408df7" containerName="cleanup"
Sep 08 16:30:55.685202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685187    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.685202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.685197    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0619e549-8978-413d-bb80-cca810a66161" containerName="istio-proxy"
Sep 08 16:30:55.698197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.698173    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.701878 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.701854    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aeb37378-openshift-default-dockercfg-4n7hd\""
Sep 08 16:30:55.702891 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.702866    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:30:55.733199 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.733169    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:55.830092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tf5r\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830364    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.830496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.830404    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tf5r\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.931743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931699    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.931838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932207 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.932115    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.932219    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932318 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.932271    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.932343    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.932446 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.932428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.934127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.934095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.934354 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.934334    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.939722 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.939666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:55.940110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:55.940089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tf5r\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r\") pod \"e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") " pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:56.010233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.010209    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:56.289374 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.289347    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:30:56.290930 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:30:56.290901    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9efd3e8_2d8a_467c_92c9_f3e95134bea2.slice/crio-55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f WatchSource:0}: Error finding container 55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f: Status 404 returned error can't find the container with id 55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f
Sep 08 16:30:56.293193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.293161    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:56.293279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.293228    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:56.293279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.293257    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:30:56.450557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.450518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerStarted","Data":"d5aa07d5804ddfc39d1e8e613c94f89d42f8fd10f4c2b9a8ed08298a9fee9ab3"}
Sep 08 16:30:56.450730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.450584    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerStarted","Data":"55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f"}
Sep 08 16:30:56.472095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.472045    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podStartSLOduration=1.472028993 podStartE2EDuration="1.472028993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:56.471032213 +0000 UTC m=+1086.115108905" watchObservedRunningTime="2026-09-08 16:30:56.472028993 +0000 UTC m=+1086.116105684"
Sep 08 16:30:56.503890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:56.503850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:57.010417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.010379    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:57.015605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.015580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:57.455422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.455391    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:57.456351 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.456333    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:30:57.852158 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.852110    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:57.955305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:57.955270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:59.201244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:59.201209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:59.456837 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:59.456735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:59.954496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:30:59.954427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:00.060909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:00.060876    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:00.136791 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:00.136739    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:31:19.077516 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:19.077469    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:31:21.580640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.580603    2571 generic.go:356] "Generic (PLEG): container finished" podID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerID="d5aa07d5804ddfc39d1e8e613c94f89d42f8fd10f4c2b9a8ed08298a9fee9ab3" exitCode=0
Sep 08 16:31:21.581187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.580684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerDied","Data":"d5aa07d5804ddfc39d1e8e613c94f89d42f8fd10f4c2b9a8ed08298a9fee9ab3"}
Sep 08 16:31:21.581271 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.581210    2571 scope.go:117] "RemoveContainer" containerID="d5aa07d5804ddfc39d1e8e613c94f89d42f8fd10f4c2b9a8ed08298a9fee9ab3"
Sep 08 16:31:21.582397 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.582365    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:31:21.582472 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.582445    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:31:21.582508 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:21.582480    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:31:22.589548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:22.589506    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerStarted","Data":"83d8520d381ccd02e316dcaefaaecd099485551067117ade3af5e428a38e800e"}
Sep 08 16:31:23.011227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:23.011126    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:23.012380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:23.012352    2571 patch_prober.go:28] interesting pod/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 16:31:23.012511 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:23.012403    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 16:31:24.010853 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:24.010820    2571 patch_prober.go:28] interesting pod/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 16:31:24.011237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:24.010873    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 16:31:25.011254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:25.011215    2571 patch_prober.go:28] interesting pod/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 16:31:25.011699 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:25.011270    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 16:31:26.011020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:26.010978    2571 patch_prober.go:28] interesting pod/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 16:31:26.011270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:26.011041    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 16:31:27.011545 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:27.011508    2571 patch_prober.go:28] interesting pod/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 16:31:27.011957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:27.011568    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 16:31:27.456103 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:27.456066    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:28.014745 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:28.014716    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:28.015597 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:28.015580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:28.805172 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:28.805143    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:29.805211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:29.805170    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:29.944138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:29.944099    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:30.411404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:30.411353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:30.497361 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:30.497299    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:31.428700 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:31.428669    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:31:31.429092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:31.428898    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy" containerID="cri-o://83d8520d381ccd02e316dcaefaaecd099485551067117ade3af5e428a38e800e" gracePeriod=30
Sep 08 16:31:31.460950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:31.460920    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:31.770118 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:31.770035    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:31:31.801231 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:31.801197    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:32.118795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:32.118749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:32.216746 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:32.216714    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:32.670697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:32.670659    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:34.728980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:34.728943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:34.828098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:34.828068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:35.857284 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:35.857248    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:31:35.860547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:35.860526    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:35.863093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:35.863072    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1b78b587-serving-cert\""
Sep 08 16:31:35.873555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:35.873500    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:31:35.916054 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:35.916016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:36.012971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.012938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.013185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.013063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.013185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.013122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fn4z\" (UniqueName: \"kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.013185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.013150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.020554 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.020523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:36.114282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.114222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.114282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.114271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fn4z\" (UniqueName: \"kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.114456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.114297    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.114456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.114346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.114456 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:31:36.114365    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-serving-cert: secret "maas-api-e2e-derive-1b78b587-serving-cert" not found
Sep 08 16:31:36.114456 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:31:36.114426    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls podName:943ebb42-55d3-4099-934f-06e5f971faf3 nodeName:}" failed. No retries permitted until 2026-09-08 16:31:36.614406958 +0000 UTC m=+1126.258483647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls") pod "maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" (UID: "943ebb42-55d3-4099-934f-06e5f971faf3") : secret "maas-api-e2e-derive-1b78b587-serving-cert" not found
Sep 08 16:31:36.115038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.115019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.117047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.117026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.125001 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.124979    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fn4z\" (UniqueName: \"kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.396497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.396416    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:31:36.413306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.413279    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:36.517684 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.517646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:36.618285 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.618251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.620572 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.620523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:36.621578 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.621541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") pod \"maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.668110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.668087    2571 generic.go:356] "Generic (PLEG): container finished" podID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerID="83d8520d381ccd02e316dcaefaaecd099485551067117ade3af5e428a38e800e" exitCode=0
Sep 08 16:31:36.668210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.668161    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerDied","Data":"83d8520d381ccd02e316dcaefaaecd099485551067117ade3af5e428a38e800e"}
Sep 08 16:31:36.668280 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.668264    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q" event={"ID":"d9efd3e8-2d8a-467c-92c9-f3e95134bea2","Type":"ContainerDied","Data":"55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f"}
Sep 08 16:31:36.668316 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.668298    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55cbc038fbc3f9da5048fcedc2ae538da40c04464bcba65c7672ea23f2fff95f"
Sep 08 16:31:36.668354 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.668319    2571 scope.go:117] "RemoveContainer" containerID="d5aa07d5804ddfc39d1e8e613c94f89d42f8fd10f4c2b9a8ed08298a9fee9ab3"
Sep 08 16:31:36.673869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.673845    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:36.772581 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.772545    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:36.820752 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.820719    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.820952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.820805    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.820952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.820847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.820952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.820944    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.820981    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tf5r\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821010    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821040    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:36.821120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821055    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821136    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821203    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket\") pod \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\" (UID: \"d9efd3e8-2d8a-467c-92c9-f3e95134bea2\") "
Sep 08 16:31:36.821750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821596    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.821913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821815    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:36.821913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821824    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:36.821913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.821895    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:36.822086 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.822055    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:36.823586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.823562    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:36.823662 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.823644    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:31:36.824020 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.823922    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:36.824185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.824166    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r" (OuterVolumeSpecName: "kube-api-access-9tf5r") pod "d9efd3e8-2d8a-467c-92c9-f3e95134bea2" (UID: "d9efd3e8-2d8a-467c-92c9-f3e95134bea2"). InnerVolumeSpecName "kube-api-access-9tf5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:36.922720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922650    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.922720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922679    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tf5r\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-kube-api-access-9tf5r\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.922720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922693    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.922720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922706    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.922720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922719    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.923144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922730    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.923144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922743    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:36.923144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:36.922755    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9efd3e8-2d8a-467c-92c9-f3e95134bea2-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:37.063062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.063039    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:31:37.065510 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:31:37.065483    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943ebb42_55d3_4099_934f_06e5f971faf3.slice/crio-7da656ce082146f217f62a3b3e9b3cf9cdb96a2539f8b764df819791c151af30 WatchSource:0}: Error finding container 7da656ce082146f217f62a3b3e9b3cf9cdb96a2539f8b764df819791c151af30: Status 404 returned error can't find the container with id 7da656ce082146f217f62a3b3e9b3cf9cdb96a2539f8b764df819791c151af30
Sep 08 16:31:37.675232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.675145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" event={"ID":"943ebb42-55d3-4099-934f-06e5f971faf3","Type":"ContainerStarted","Data":"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"}
Sep 08 16:31:37.675232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.675207    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" event={"ID":"943ebb42-55d3-4099-934f-06e5f971faf3","Type":"ContainerStarted","Data":"7da656ce082146f217f62a3b3e9b3cf9cdb96a2539f8b764df819791c151af30"}
Sep 08 16:31:37.675426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.675241    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:37.676607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.676585    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"
Sep 08 16:31:37.695873 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.695830    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" podStartSLOduration=2.342978553 podStartE2EDuration="2.695815379s" totalImagesPullingTime="352.836826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:37.694230301 +0000 UTC m=+1127.338306995" watchObservedRunningTime="2026-09-08 16:31:37.695815379 +0000 UTC m=+1127.339892090"
Sep 08 16:31:37.732223 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.732190    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:31:37.739520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:37.739487    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aeb37378-openshift-default-ff7f7bf5b-xwd8q"]
Sep 08 16:31:38.887691 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:38.887664    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" path="/var/lib/kubelet/pods/d9efd3e8-2d8a-467c-92c9-f3e95134bea2/volumes"
Sep 08 16:31:43.686401 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:43.686372    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:31:46.790263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:46.790208    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" podUID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" containerName="main" containerID="cri-o://b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47" gracePeriod=45
Sep 08 16:31:47.044317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.044260    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:31:47.216548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216512    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216693 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216563    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216693 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216693 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216650    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216693 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdnmx\" (UniqueName: \"kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216805    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs\") pod \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\" (UID: \"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef\") "
Sep 08 16:31:47.216992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216941    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home" (OuterVolumeSpecName: "home") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:47.216992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.216958    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:47.217107 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.217030    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:47.217533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.217307    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.217533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.217494    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.217533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.217509    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.219144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.219113    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm" (OuterVolumeSpecName: "dshm") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:47.219144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.219119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx" (OuterVolumeSpecName: "kube-api-access-kdnmx") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "kube-api-access-kdnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:47.219462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.219446    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" (UID: "13c7cf28-a46d-4d6a-9419-7b1ee0b414ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:31:47.318404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.318377    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.318404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.318403    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnmx\" (UniqueName: \"kubernetes.io/projected/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-kube-api-access-kdnmx\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.318573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.318413    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:31:47.389304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389277    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:31:47.389692 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389675    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" containerName="main"
Sep 08 16:31:47.389784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389695    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" containerName="main"
Sep 08 16:31:47.389784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389718    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.389784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389726    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.389784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389737    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.389784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389743    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.390046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389815    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.390046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389822    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9efd3e8-2d8a-467c-92c9-f3e95134bea2" containerName="istio-proxy"
Sep 08 16:31:47.390046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.389828    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" containerName="main"
Sep 08 16:31:47.391986 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.391968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.396092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.396072    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d9aa4\"/\"test-model-0d9aa4-kserve-self-signed-certs\""
Sep 08 16:31:47.397233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.397210    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d9aa4\"/\"openshift-service-ca.crt\""
Sep 08 16:31:47.397393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.397324    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d9aa4\"/\"default-dockercfg-ms8qs\""
Sep 08 16:31:47.397393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.397384    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d9aa4\"/\"kube-root-ca.crt\""
Sep 08 16:31:47.404135 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.404110    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:31:47.434423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.434398    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:47.519905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.519875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.520049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.519922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpv4\" (UniqueName: \"kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.520049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.519949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.520049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.519988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.520049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.520010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.520049 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.520031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621112 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621112 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpv4\" (UniqueName: \"kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621525 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.621595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.622026 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.621912    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.623655 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.623634    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.624099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.624077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.629725 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.629698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpv4\" (UniqueName: \"kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4\") pod \"test-model-0d9aa4-kserve-697cb97567-9f9pv\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") " pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.702887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.702842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:47.730340 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.730309    2571 generic.go:356] "Generic (PLEG): container finished" podID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" containerID="b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47" exitCode=0
Sep 08 16:31:47.730528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.730369    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"
Sep 08 16:31:47.730528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.730399    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" event={"ID":"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef","Type":"ContainerDied","Data":"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"}
Sep 08 16:31:47.730528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.730455    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57" event={"ID":"13c7cf28-a46d-4d6a-9419-7b1ee0b414ef","Type":"ContainerDied","Data":"f04900b14f6cf005e7aa225a55d96d5c9c6bd51082b0e900049894fdbd006e4c"}
Sep 08 16:31:47.730528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.730485    2571 scope.go:117] "RemoveContainer" containerID="b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"
Sep 08 16:31:47.740113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.740093    2571 scope.go:117] "RemoveContainer" containerID="b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"
Sep 08 16:31:47.740380 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:31:47.740363    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47\": container with ID starting with b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47 not found: ID does not exist" containerID="b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"
Sep 08 16:31:47.740445 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.740387    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47"} err="failed to get container status \"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47\": rpc error: code = NotFound desc = could not find container \"b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47\": container with ID starting with b31a7dd128478e87a8189710030a50f993020db39e5407569793d163c6a57f47 not found: ID does not exist"
Sep 08 16:31:47.754054 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.754030    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:31:47.764047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.763995    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-ec6a8395-kserve-75c54b5b99-tlv57"]
Sep 08 16:31:47.982911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:47.982785    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:31:47.985528 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:31:47.985500    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcb740f2_bd33_4fe1_8c08_02c68d1dede5.slice/crio-a2e7114485634857c4f375e2f391a7453abd2c2430f52b66f22308a3080d5ff6 WatchSource:0}: Error finding container a2e7114485634857c4f375e2f391a7453abd2c2430f52b66f22308a3080d5ff6: Status 404 returned error can't find the container with id a2e7114485634857c4f375e2f391a7453abd2c2430f52b66f22308a3080d5ff6
Sep 08 16:31:48.737316 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:48.737281    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" event={"ID":"fcb740f2-bd33-4fe1-8c08-02c68d1dede5","Type":"ContainerStarted","Data":"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"}
Sep 08 16:31:48.737505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:48.737333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" event={"ID":"fcb740f2-bd33-4fe1-8c08-02c68d1dede5","Type":"ContainerStarted","Data":"a2e7114485634857c4f375e2f391a7453abd2c2430f52b66f22308a3080d5ff6"}
Sep 08 16:31:48.754445 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:48.754400    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" podStartSLOduration=1.754386386 podStartE2EDuration="1.754386386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:48.752901739 +0000 UTC m=+1138.396978431" watchObservedRunningTime="2026-09-08 16:31:48.754386386 +0000 UTC m=+1138.398463077"
Sep 08 16:31:48.888590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:48.888556    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c7cf28-a46d-4d6a-9419-7b1ee0b414ef" path="/var/lib/kubelet/pods/13c7cf28-a46d-4d6a-9419-7b1ee0b414ef/volumes"
Sep 08 16:31:50.594018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:50.593982    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:31:54.879526 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:54.879491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:31:57.703799 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:57.703742    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:57.704288 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:57.703826    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:57.716124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:57.716102    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:31:57.794858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:31:57.794832    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:32:02.269990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:02.269954    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:06.131554 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:06.131514    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:06.880129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:06.880093    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:32:06.880444 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:06.880388    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" podUID="943ebb42-55d3-4099-934f-06e5f971faf3" containerName="maas-api" containerID="cri-o://8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7" gracePeriod=30
Sep 08 16:32:06.930608 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:06.930578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:07.072832 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.072801    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:07.126285 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.126261    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:32:07.196939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.196878    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs\") pod \"943ebb42-55d3-4099-934f-06e5f971faf3\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") "
Sep 08 16:32:07.196939 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.196909    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca\") pod \"943ebb42-55d3-4099-934f-06e5f971faf3\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") "
Sep 08 16:32:07.197296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.196977    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") pod \"943ebb42-55d3-4099-934f-06e5f971faf3\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") "
Sep 08 16:32:07.197296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.197076    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fn4z\" (UniqueName: \"kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z\") pod \"943ebb42-55d3-4099-934f-06e5f971faf3\" (UID: \"943ebb42-55d3-4099-934f-06e5f971faf3\") "
Sep 08 16:32:07.197382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.197329    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "943ebb42-55d3-4099-934f-06e5f971faf3" (UID: "943ebb42-55d3-4099-934f-06e5f971faf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:32:07.197438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.197403    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943ebb42-55d3-4099-934f-06e5f971faf3-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.199263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.199237    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "943ebb42-55d3-4099-934f-06e5f971faf3" (UID: "943ebb42-55d3-4099-934f-06e5f971faf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:32:07.199365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.199262    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "943ebb42-55d3-4099-934f-06e5f971faf3" (UID: "943ebb42-55d3-4099-934f-06e5f971faf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:32:07.199365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.199267    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z" (OuterVolumeSpecName: "kube-api-access-6fn4z") pod "943ebb42-55d3-4099-934f-06e5f971faf3" (UID: "943ebb42-55d3-4099-934f-06e5f971faf3"). InnerVolumeSpecName "kube-api-access-6fn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:32:07.296478 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.296441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:07.298574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.298550    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fn4z\" (UniqueName: \"kubernetes.io/projected/943ebb42-55d3-4099-934f-06e5f971faf3-kube-api-access-6fn4z\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.298574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.298576    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.298751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.298590    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943ebb42-55d3-4099-934f-06e5f971faf3-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.406427 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.406390    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:07.521702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.521620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:07.834294 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.834259    2571 generic.go:356] "Generic (PLEG): container finished" podID="943ebb42-55d3-4099-934f-06e5f971faf3" containerID="8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7" exitCode=0
Sep 08 16:32:07.834487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.834310    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"
Sep 08 16:32:07.834487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.834345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" event={"ID":"943ebb42-55d3-4099-934f-06e5f971faf3","Type":"ContainerDied","Data":"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"}
Sep 08 16:32:07.834487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.834434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm" event={"ID":"943ebb42-55d3-4099-934f-06e5f971faf3","Type":"ContainerDied","Data":"7da656ce082146f217f62a3b3e9b3cf9cdb96a2539f8b764df819791c151af30"}
Sep 08 16:32:07.834487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.834465    2571 scope.go:117] "RemoveContainer" containerID="8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"
Sep 08 16:32:07.844119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.844103    2571 scope.go:117] "RemoveContainer" containerID="8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"
Sep 08 16:32:07.844385 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:32:07.844366    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7\": container with ID starting with 8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7 not found: ID does not exist" containerID="8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"
Sep 08 16:32:07.844431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.844393    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7"} err="failed to get container status \"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7\": rpc error: code = NotFound desc = could not find container \"8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7\": container with ID starting with 8c672e26036f864a0766c0a53ddd31566e85445b8207735c9396f1e859d213e7 not found: ID does not exist"
Sep 08 16:32:07.858750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.858731    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:32:07.864448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:07.864428    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b78b587-6566f8ddf4-cgttm"]
Sep 08 16:32:08.210293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:08.210202    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:08.307116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:08.307079    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:08.888118 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:08.888086    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="943ebb42-55d3-4099-934f-06e5f971faf3" path="/var/lib/kubelet/pods/943ebb42-55d3-4099-934f-06e5f971faf3/volumes"
Sep 08 16:32:09.733890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:09.733860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:13.292923 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:13.292808    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:32:15.281899 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.281867    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:32:15.282268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.282255    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943ebb42-55d3-4099-934f-06e5f971faf3" containerName="maas-api"
Sep 08 16:32:15.282268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.282269    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="943ebb42-55d3-4099-934f-06e5f971faf3" containerName="maas-api"
Sep 08 16:32:15.282344 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.282324    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="943ebb42-55d3-4099-934f-06e5f971faf3" containerName="maas-api"
Sep 08 16:32:15.285025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.285004    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.288561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.288538    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b41034\"/\"kube-root-ca.crt\""
Sep 08 16:32:15.288687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.288544    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b41034\"/\"test-model-b41034-kserve-self-signed-certs\""
Sep 08 16:32:15.288687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.288593    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b41034\"/\"default-dockercfg-ljfwn\""
Sep 08 16:32:15.288687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.288561    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b41034\"/\"openshift-service-ca.crt\""
Sep 08 16:32:15.293932 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.293910    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:32:15.310450 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.310427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:15.365637 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.365756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.365756 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.365872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnth\" (UniqueName: \"kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.365872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365834    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.365872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.365860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.466995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.466968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467047    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467108    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnth\" (UniqueName: \"kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467485    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.467590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.467569    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.469495 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.469472    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.469670 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.469655    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.475891 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.475866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnth\" (UniqueName: \"kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth\") pod \"test-model-b41034-kserve-7c64f6b8fc-x48c2\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") " pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.595004 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.594965    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:15.880085 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:15.880061    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:32:15.882240 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:32:15.882209    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b0bc80_114c_44da_8c7d_d97b5cd1414c.slice/crio-4fa19d856fe11745a94472771261440aeade00fe81d73bff8b4e3dfe890a5f42 WatchSource:0}: Error finding container 4fa19d856fe11745a94472771261440aeade00fe81d73bff8b4e3dfe890a5f42: Status 404 returned error can't find the container with id 4fa19d856fe11745a94472771261440aeade00fe81d73bff8b4e3dfe890a5f42
Sep 08 16:32:16.114169 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.114130    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:16.229936 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.229857    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:16.334886 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.334855    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:16.433880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.433852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:16.825242 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.825205    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:16.881120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.881092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" event={"ID":"e1b0bc80-114c-44da-8c7d-d97b5cd1414c","Type":"ContainerStarted","Data":"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"}
Sep 08 16:32:16.881254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.881132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" event={"ID":"e1b0bc80-114c-44da-8c7d-d97b5cd1414c","Type":"ContainerStarted","Data":"4fa19d856fe11745a94472771261440aeade00fe81d73bff8b4e3dfe890a5f42"}
Sep 08 16:32:16.901626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:16.901583    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" podStartSLOduration=1.901568283 podStartE2EDuration="1.901568283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:32:16.899355816 +0000 UTC m=+1166.543432508" watchObservedRunningTime="2026-09-08 16:32:16.901568283 +0000 UTC m=+1166.545644975"
Sep 08 16:32:18.503153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:18.503121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:25.595202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:25.595165    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:25.595601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:25.595215    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:25.608315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:25.608286    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:25.934510 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:25.934435    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:32:33.379416 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:33.379376    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:32:42.694076 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:42.694036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:46.702363 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:46.702321    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:46.803129 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:46.803097    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:48.277034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:48.276998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:49.265349 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:49.265315    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:49.400590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:49.400559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:49.525244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:49.525135    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:32:50.876123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:50.876100    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:32:50.876664 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:50.876648    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:32:51.284420 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:32:51.284335    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:05.342144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:05.342085    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:07.173991 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:07.173960    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:11.274324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:11.274290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:11.384815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:11.384749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:11.503333 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:11.503300    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:19.699538 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:19.699492    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:20.164414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:20.164380    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:33:21.178906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:21.178872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:21.291535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:21.291501    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:33:21.312073 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:21.312035    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:21.633717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:21.633668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:21.956655 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:21.956563    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:22.666851 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:22.666825    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:25.097456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:25.097421    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:28.563919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:28.563887    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:28.665957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:28.665924    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:28.772266 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:28.772231    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:28.944844 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:28.944814    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:29.516517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:29.516486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:29.759184 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:29.759135    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:31.095469 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:31.095422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:31.958200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:31.958171    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:33:31.958414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:31.958390    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" podUID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" containerName="maas-api" containerID="cri-o://5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a" gracePeriod=30
Sep 08 16:33:31.989191 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:31.989158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:32.092988 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.092956    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:32.194968 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.194935    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:32.293164 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.293132    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:32.411372 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.411350    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:33:32.492153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492079    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca\") pod \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") "
Sep 08 16:33:32.492295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492175    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs\") pod \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") "
Sep 08 16:33:32.492295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492214    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfc6z\" (UniqueName: \"kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z\") pod \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") "
Sep 08 16:33:32.492295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492252    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") pod \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\" (UID: \"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d\") "
Sep 08 16:33:32.492453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492426    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" (UID: "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:32.492570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.492554    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.494386 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.494365    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" (UID: "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:32.494386 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.494372    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z" (OuterVolumeSpecName: "kube-api-access-qfc6z") pod "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" (UID: "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d"). InnerVolumeSpecName "kube-api-access-qfc6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:32.494490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.494475    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" (UID: "d2eeeca7-61e8-48e4-8833-3afb9fb60a5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:32.593409 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.593379    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.593409 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.593406    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfc6z\" (UniqueName: \"kubernetes.io/projected/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-kube-api-access-qfc6z\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.593568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.593416    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.627079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:32.627055    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:33.250595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.250512    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" containerID="5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a" exitCode=0
Sep 08 16:33:33.250595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.250575    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"
Sep 08 16:33:33.251084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.250590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" event={"ID":"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d","Type":"ContainerDied","Data":"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"}
Sep 08 16:33:33.251084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.250650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6" event={"ID":"d2eeeca7-61e8-48e4-8833-3afb9fb60a5d","Type":"ContainerDied","Data":"f7e54b703117b24ac962eca48354601513587077e9d5982e6653030a0515bcb0"}
Sep 08 16:33:33.251084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.250683    2571 scope.go:117] "RemoveContainer" containerID="5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"
Sep 08 16:33:33.259548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.259531    2571 scope.go:117] "RemoveContainer" containerID="5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"
Sep 08 16:33:33.259838 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:33.259814    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a\": container with ID starting with 5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a not found: ID does not exist" containerID="5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"
Sep 08 16:33:33.259933 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.259844    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a"} err="failed to get container status \"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a\": rpc error: code = NotFound desc = could not find container \"5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a\": container with ID starting with 5e97303356d57c58b21b4e65c02fe32287bdbae6c2923d235f681209a00b470a not found: ID does not exist"
Sep 08 16:33:33.270145 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.270124    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:33:33.276583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:33.276564    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9aa4-5654b6ddb9-dt8k6"]
Sep 08 16:33:34.857873 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:34.857841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:34.888012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:34.887980    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" path="/var/lib/kubelet/pods/d2eeeca7-61e8-48e4-8833-3afb9fb60a5d/volumes"
Sep 08 16:33:35.090564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.090524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:35.184793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.184692    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" podUID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" containerName="main" containerID="cri-o://4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb" gracePeriod=45
Sep 08 16:33:35.431402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.431376    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:33:35.483030 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.482951    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:35.522750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522719    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.522911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522785    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.522911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522825    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.522911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522884    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.522911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522911    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.523181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.522980    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxpv4\" (UniqueName: \"kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4\") pod \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\" (UID: \"fcb740f2-bd33-4fe1-8c08-02c68d1dede5\") "
Sep 08 16:33:35.523181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523023    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:35.523181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:35.523181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523129    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home" (OuterVolumeSpecName: "home") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:35.523426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523343    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:35.523426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523366    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:35.523426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.523379    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:35.525838 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.525812    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:35.525838 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.525822    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4" (OuterVolumeSpecName: "kube-api-access-xxpv4") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "kube-api-access-xxpv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:35.525838 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.525826    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm" (OuterVolumeSpecName: "dshm") pod "fcb740f2-bd33-4fe1-8c08-02c68d1dede5" (UID: "fcb740f2-bd33-4fe1-8c08-02c68d1dede5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:35.624442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.624410    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:35.624442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.624438    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:35.624442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:35.624447    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpv4\" (UniqueName: \"kubernetes.io/projected/fcb740f2-bd33-4fe1-8c08-02c68d1dede5-kube-api-access-xxpv4\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.269050 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.269017    2571 generic.go:356] "Generic (PLEG): container finished" podID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" containerID="4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb" exitCode=0
Sep 08 16:33:36.269593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.269077    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" event={"ID":"fcb740f2-bd33-4fe1-8c08-02c68d1dede5","Type":"ContainerDied","Data":"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"}
Sep 08 16:33:36.269593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.269082    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"
Sep 08 16:33:36.269593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.269107    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv" event={"ID":"fcb740f2-bd33-4fe1-8c08-02c68d1dede5","Type":"ContainerDied","Data":"a2e7114485634857c4f375e2f391a7453abd2c2430f52b66f22308a3080d5ff6"}
Sep 08 16:33:36.269593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.269126    2571 scope.go:117] "RemoveContainer" containerID="4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"
Sep 08 16:33:36.278128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.278112    2571 scope.go:117] "RemoveContainer" containerID="4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"
Sep 08 16:33:36.278413 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:36.278392    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb\": container with ID starting with 4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb not found: ID does not exist" containerID="4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"
Sep 08 16:33:36.278498 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.278420    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb"} err="failed to get container status \"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb\": rpc error: code = NotFound desc = could not find container \"4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb\": container with ID starting with 4cbaa0983a570472ef0820fd7cdf9f3fa385d86a515ba01e3e11cbb3645caeeb not found: ID does not exist"
Sep 08 16:33:36.293863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.293839    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:33:36.298369 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.298337    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0d9aa4/test-model-0d9aa4-kserve-697cb97567-9f9pv"]
Sep 08 16:33:36.313704 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.313658    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" podUID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" containerName="main" containerID="cri-o://c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8" gracePeriod=45
Sep 08 16:33:36.585691 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.585654    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:33:36.735829 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.735798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.735829 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.735836    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcnth\" (UniqueName: \"kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.736024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.735955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.736024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736014    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.736170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736033    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home" (OuterVolumeSpecName: "home") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:36.736170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736052    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.736170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm\") pod \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\" (UID: \"e1b0bc80-114c-44da-8c7d-d97b5cd1414c\") "
Sep 08 16:33:36.736289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736182    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:36.736289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736238    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:36.736433 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736417    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.736489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736435    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.736489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.736445    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.738188 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.738154    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth" (OuterVolumeSpecName: "kube-api-access-mcnth") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "kube-api-access-mcnth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:36.738306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.738211    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm" (OuterVolumeSpecName: "dshm") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:36.738306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.738254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1b0bc80-114c-44da-8c7d-d97b5cd1414c" (UID: "e1b0bc80-114c-44da-8c7d-d97b5cd1414c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:36.837324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.837298    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcnth\" (UniqueName: \"kubernetes.io/projected/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-kube-api-access-mcnth\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.837324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.837321    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.837324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.837330    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b0bc80-114c-44da-8c7d-d97b5cd1414c-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:36.893057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.893027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:36.893674 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.893654    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" path="/var/lib/kubelet/pods/fcb740f2-bd33-4fe1-8c08-02c68d1dede5/volumes"
Sep 08 16:33:36.994399 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:36.994362    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:37.275309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.275208    2571 generic.go:356] "Generic (PLEG): container finished" podID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" containerID="c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8" exitCode=0
Sep 08 16:33:37.275309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.275272    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"
Sep 08 16:33:37.275841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.275304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" event={"ID":"e1b0bc80-114c-44da-8c7d-d97b5cd1414c","Type":"ContainerDied","Data":"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"}
Sep 08 16:33:37.275841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.275353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2" event={"ID":"e1b0bc80-114c-44da-8c7d-d97b5cd1414c","Type":"ContainerDied","Data":"4fa19d856fe11745a94472771261440aeade00fe81d73bff8b4e3dfe890a5f42"}
Sep 08 16:33:37.275841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.275377    2571 scope.go:117] "RemoveContainer" containerID="c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"
Sep 08 16:33:37.283993 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.283973    2571 scope.go:117] "RemoveContainer" containerID="c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"
Sep 08 16:33:37.284273 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:37.284246    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8\": container with ID starting with c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8 not found: ID does not exist" containerID="c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"
Sep 08 16:33:37.284325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.284281    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8"} err="failed to get container status \"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8\": rpc error: code = NotFound desc = could not find container \"c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8\": container with ID starting with c3cf05635f2fd370b1bea5eb25b5af007f66df488fe80b1ffa650335a60859e8 not found: ID does not exist"
Sep 08 16:33:37.290734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.290708    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:33:37.295520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.295499    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b41034/test-model-b41034-kserve-7c64f6b8fc-x48c2"]
Sep 08 16:33:37.523955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:37.523921    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:38.535725 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.535688    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:38.604914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.604887    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:38.817272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.817235    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:33:38.817497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.817473    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy" containerID="cri-o://62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8" gracePeriod=30
Sep 08 16:33:38.838356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.838320    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:38.890462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:38.890433    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" path="/var/lib/kubelet/pods/e1b0bc80-114c-44da-8c7d-d97b5cd1414c/volumes"
Sep 08 16:33:39.284956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:39.284865    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:43.706060 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:43.705969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:43.807278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:43.807238    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:43.952795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:43.952747    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:44.057861 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.057839    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:33:44.207005 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.206975    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b677j\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207005 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207009    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207048    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207091    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207187    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207210    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo\") pod \"0b75000b-d506-4b79-bf9f-9dc220901f12\" (UID: \"0b75000b-d506-4b79-bf9f-9dc220901f12\") "
Sep 08 16:33:44.207452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207331    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:44.207452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207351    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:44.207669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207559    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:44.207669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207607    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.207669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207627    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.207859 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207752    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:44.207905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.207855    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:44.209395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.209377    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j" (OuterVolumeSpecName: "kube-api-access-b677j") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "kube-api-access-b677j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:44.209493 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.209419    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:33:44.209806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.209752    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:33:44.209806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.209752    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b75000b-d506-4b79-bf9f-9dc220901f12" (UID: "0b75000b-d506-4b79-bf9f-9dc220901f12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:44.260394 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260320    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:33:44.260654 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260642    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" containerName="main"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260656    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" containerName="main"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260669    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" containerName="maas-api"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260675    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" containerName="maas-api"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260684    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260689    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.260701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260702    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" containerName="main"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260708    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" containerName="main"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260719    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260723    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260798    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2eeeca7-61e8-48e4-8833-3afb9fb60a5d" containerName="maas-api"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260810    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b0bc80-114c-44da-8c7d-d97b5cd1414c" containerName="main"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260816    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260823    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb740f2-bd33-4fe1-8c08-02c68d1dede5" containerName="main"
Sep 08 16:33:44.260912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.260828    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerName="istio-proxy"
Sep 08 16:33:44.263075 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.263059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.266871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.266850    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-57c06fce-openshift-default-dockercfg-md6qm\""
Sep 08 16:33:44.279624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.279606    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:33:44.291026 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.290983    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:44.308448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308429    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b677j\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-kube-api-access-b677j\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308450    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308461    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308470    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308478    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b75000b-d506-4b79-bf9f-9dc220901f12-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308490    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b75000b-d506-4b79-bf9f-9dc220901f12-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.308546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.308501    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b75000b-d506-4b79-bf9f-9dc220901f12-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:44.314103 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.314080    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b75000b-d506-4b79-bf9f-9dc220901f12" containerID="62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8" exitCode=0
Sep 08 16:33:44.314199 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.314130    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"
Sep 08 16:33:44.314199 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.314163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerDied","Data":"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"}
Sep 08 16:33:44.314289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.314210    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m" event={"ID":"0b75000b-d506-4b79-bf9f-9dc220901f12","Type":"ContainerDied","Data":"642a214f080b0a83ba5730fdcdb8e64326f6cda7099a575bae0e80caad986649"}
Sep 08 16:33:44.314289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.314232    2571 scope.go:117] "RemoveContainer" containerID="62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"
Sep 08 16:33:44.323094 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.323078    2571 scope.go:117] "RemoveContainer" containerID="1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"
Sep 08 16:33:44.330813 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.330794    2571 scope.go:117] "RemoveContainer" containerID="62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"
Sep 08 16:33:44.331049 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:44.331033    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8\": container with ID starting with 62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8 not found: ID does not exist" containerID="62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"
Sep 08 16:33:44.331101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.331056    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8"} err="failed to get container status \"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8\": rpc error: code = NotFound desc = could not find container \"62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8\": container with ID starting with 62512ab27ef07fb14bc465d9cf50fa3432ab96ae256a4fd225a5a7a3e7933aa8 not found: ID does not exist"
Sep 08 16:33:44.331101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.331071    2571 scope.go:117] "RemoveContainer" containerID="1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"
Sep 08 16:33:44.331295 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:44.331278    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1\": container with ID starting with 1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1 not found: ID does not exist" containerID="1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"
Sep 08 16:33:44.331335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.331304    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1"} err="failed to get container status \"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1\": rpc error: code = NotFound desc = could not find container \"1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1\": container with ID starting with 1ffa0c94fef85c8c32cf0c779d11c2ba4ef19665fc7706193a76907aa3322be1 not found: ID does not exist"
Sep 08 16:33:44.336869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.336849    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:33:44.341533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.341509    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9aa4-openshift-default-5b7c8f6c4d-t4g5m"]
Sep 08 16:33:44.409464 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409432    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409496    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409587    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv95\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.409949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.409744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.510966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511008    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv95\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511338 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511616 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511616 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511391    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511616 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511398    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.511855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511810    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.512186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.511877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.513706 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.513677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.513946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.513929    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.523330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.523302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.523438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.523358    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv95\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95\") pod \"e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") " pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.538159 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.538133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:44.573200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.573179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:44.853456 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.853425    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:33:44.855583 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:33:44.855561    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d68f6a7_31f3_4443_9584_c35231c9c9d6.slice/crio-71104f3a91255d41840af26b1460b1b53eee7a05bbf017253fe2c77411a81991 WatchSource:0}: Error finding container 71104f3a91255d41840af26b1460b1b53eee7a05bbf017253fe2c77411a81991: Status 404 returned error can't find the container with id 71104f3a91255d41840af26b1460b1b53eee7a05bbf017253fe2c77411a81991
Sep 08 16:33:44.857682 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.857651    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:44.857786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.857716    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:44.857786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.857744    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:44.887755 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:44.887727    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b75000b-d506-4b79-bf9f-9dc220901f12" path="/var/lib/kubelet/pods/0b75000b-d506-4b79-bf9f-9dc220901f12/volumes"
Sep 08 16:33:45.170355 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.170268    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:45.321313 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.321273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerStarted","Data":"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"}
Sep 08 16:33:45.321517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.321330    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerStarted","Data":"71104f3a91255d41840af26b1460b1b53eee7a05bbf017253fe2c77411a81991"}
Sep 08 16:33:45.346183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.346126    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podStartSLOduration=1.346109913 podStartE2EDuration="1.346109913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:45.344901281 +0000 UTC m=+1254.988977972" watchObservedRunningTime="2026-09-08 16:33:45.346109913 +0000 UTC m=+1254.990186604"
Sep 08 16:33:45.574062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.574034    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:45.579076 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.579051    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:45.772613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.772414    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"]
Sep 08 16:33:45.774276 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.774249    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-293166-29814750-6gksl"]
Sep 08 16:33:45.775287 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.775262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:45.826647 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.826574    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:33:45.826866 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.826841    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" podUID="ecb62fdf-d711-41bc-a139-851d42772008" containerName="payload-processing" containerID="cri-o://874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682" gracePeriod=30
Sep 08 16:33:45.918565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:45.918530    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:46.052326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.052283    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:46.077966 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.077910    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:33:46.116858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.116821    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:46.122870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.122847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume\") pod \"ecb62fdf-d711-41bc-a139-851d42772008\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") "
Sep 08 16:33:46.122962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.122918    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzm2l\" (UniqueName: \"kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l\") pod \"ecb62fdf-d711-41bc-a139-851d42772008\" (UID: \"ecb62fdf-d711-41bc-a139-851d42772008\") "
Sep 08 16:33:46.123250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.123224    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecb62fdf-d711-41bc-a139-851d42772008" (UID: "ecb62fdf-d711-41bc-a139-851d42772008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:46.125157 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.125135    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l" (OuterVolumeSpecName: "kube-api-access-jzm2l") pod "ecb62fdf-d711-41bc-a139-851d42772008" (UID: "ecb62fdf-d711-41bc-a139-851d42772008"). InnerVolumeSpecName "kube-api-access-jzm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:46.224412 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.224381    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb62fdf-d711-41bc-a139-851d42772008-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:46.224412 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.224407    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzm2l\" (UniqueName: \"kubernetes.io/projected/ecb62fdf-d711-41bc-a139-851d42772008-kube-api-access-jzm2l\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:46.328273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328181    2571 generic.go:356] "Generic (PLEG): container finished" podID="ecb62fdf-d711-41bc-a139-851d42772008" containerID="874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682" exitCode=0
Sep 08 16:33:46.328273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328229    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"
Sep 08 16:33:46.328273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328260    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" event={"ID":"ecb62fdf-d711-41bc-a139-851d42772008","Type":"ContainerDied","Data":"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"}
Sep 08 16:33:46.328497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn" event={"ID":"ecb62fdf-d711-41bc-a139-851d42772008","Type":"ContainerDied","Data":"88750e495b0f66a1f34f0db4d18dfee769502674fdf77e4ebe7deee6f3a29da7"}
Sep 08 16:33:46.328497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328331    2571 scope.go:117] "RemoveContainer" containerID="874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"
Sep 08 16:33:46.328808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.328758    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:46.329908 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.329885    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:33:46.338352 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.338330    2571 scope.go:117] "RemoveContainer" containerID="874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"
Sep 08 16:33:46.338696 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:46.338674    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682\": container with ID starting with 874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682 not found: ID does not exist" containerID="874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"
Sep 08 16:33:46.338747 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.338704    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682"} err="failed to get container status \"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682\": rpc error: code = NotFound desc = could not find container \"874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682\": container with ID starting with 874ad5caec8fef321f32926726e58f65cb0a75be787b992f1b1bd0ea9af0b682 not found: ID does not exist"
Sep 08 16:33:46.369219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.368530    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:33:46.380018 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.379981    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-293166-545989b9dc-4xgrn"]
Sep 08 16:33:46.455439 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.455400    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:46.581519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.581439    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:46.888163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.888086    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1c0e75-f5e4-40dd-97ba-0a75db408df7" path="/var/lib/kubelet/pods/3a1c0e75-f5e4-40dd-97ba-0a75db408df7/volumes"
Sep 08 16:33:46.888414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:46.888402    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb62fdf-d711-41bc-a139-851d42772008" path="/var/lib/kubelet/pods/ecb62fdf-d711-41bc-a139-851d42772008/volumes"
Sep 08 16:33:47.317347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:47.317316    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:47.426882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:47.426853    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:48.502410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.502373    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:33:48.502934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.502917    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb62fdf-d711-41bc-a139-851d42772008" containerName="payload-processing"
Sep 08 16:33:48.502989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.502937    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb62fdf-d711-41bc-a139-851d42772008" containerName="payload-processing"
Sep 08 16:33:48.503064 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.503051    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb62fdf-d711-41bc-a139-851d42772008" containerName="payload-processing"
Sep 08 16:33:48.506196 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.506181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.510289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.510267    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-57c06fce-serving-cert\""
Sep 08 16:33:48.516916 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.516895    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:33:48.548730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.548698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgvt\" (UniqueName: \"kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.548876 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.548758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.548922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.548888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.548958 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.548940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.557238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.557216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:48.650188 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.650152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.650385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.650214    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.650385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.650250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgvt\" (UniqueName: \"kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.650385 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:48.650315    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-serving-cert: secret "maas-api-e2e-mt-57c06fce-serving-cert" not found
Sep 08 16:33:48.650385 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.650316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.650385 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:48.650384    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls podName:12f0c6ae-3e9c-4fca-b938-1a18a343640a nodeName:}" failed. No retries permitted until 2026-09-08 16:33:49.150364903 +0000 UTC m=+1258.794441575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls") pod "maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a") : secret "maas-api-e2e-mt-57c06fce-serving-cert" not found
Sep 08 16:33:48.651069 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.651042    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:48.651227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.651208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.652902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.652884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:48.670428 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:48.670404    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgvt\" (UniqueName: \"kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:49.154255 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.154218    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:49.154453 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:49.154369    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-serving-cert: secret "maas-api-e2e-mt-57c06fce-serving-cert" not found
Sep 08 16:33:49.154517 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:49.154461    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls podName:12f0c6ae-3e9c-4fca-b938-1a18a343640a nodeName:}" failed. No retries permitted until 2026-09-08 16:33:50.154438812 +0000 UTC m=+1259.798515500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls") pod "maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a") : secret "maas-api-e2e-mt-57c06fce-serving-cert" not found
Sep 08 16:33:49.160936 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.160900    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:49.256097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.256065    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:49.489002 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.488915    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:49.489645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.489621    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:33:49.489898 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.489874    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" podUID="606ef2f4-b8c4-4907-9698-67d78cb0456f" containerName="maas-api" containerID="cri-o://fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a" gracePeriod=30
Sep 08 16:33:49.733451 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.733430    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:33:49.760208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.760153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") pod \"606ef2f4-b8c4-4907-9698-67d78cb0456f\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") "
Sep 08 16:33:49.760208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.760195    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88ncl\" (UniqueName: \"kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl\") pod \"606ef2f4-b8c4-4907-9698-67d78cb0456f\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") "
Sep 08 16:33:49.760395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.760319    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca\") pod \"606ef2f4-b8c4-4907-9698-67d78cb0456f\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") "
Sep 08 16:33:49.760395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.760354    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs\") pod \"606ef2f4-b8c4-4907-9698-67d78cb0456f\" (UID: \"606ef2f4-b8c4-4907-9698-67d78cb0456f\") "
Sep 08 16:33:49.760663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.760638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "606ef2f4-b8c4-4907-9698-67d78cb0456f" (UID: "606ef2f4-b8c4-4907-9698-67d78cb0456f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:49.762598 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.762570    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl" (OuterVolumeSpecName: "kube-api-access-88ncl") pod "606ef2f4-b8c4-4907-9698-67d78cb0456f" (UID: "606ef2f4-b8c4-4907-9698-67d78cb0456f"). InnerVolumeSpecName "kube-api-access-88ncl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:49.762906 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.762881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "606ef2f4-b8c4-4907-9698-67d78cb0456f" (UID: "606ef2f4-b8c4-4907-9698-67d78cb0456f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:49.762993 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.762969    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "606ef2f4-b8c4-4907-9698-67d78cb0456f" (UID: "606ef2f4-b8c4-4907-9698-67d78cb0456f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:33:49.861380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.861339    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ef2f4-b8c4-4907-9698-67d78cb0456f-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:49.861380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.861372    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:49.861380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.861382    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ef2f4-b8c4-4907-9698-67d78cb0456f-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:49.861380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:49.861390    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ncl\" (UniqueName: \"kubernetes.io/projected/606ef2f4-b8c4-4907-9698-67d78cb0456f-kube-api-access-88ncl\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:33:50.131606 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.131574    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:50.163701 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.163668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:50.166277 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.166258    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:50.204902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.204874    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:50.318580 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.318548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:50.351062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.351025    2571 generic.go:356] "Generic (PLEG): container finished" podID="606ef2f4-b8c4-4907-9698-67d78cb0456f" containerID="fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a" exitCode=0
Sep 08 16:33:50.351240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.351070    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"
Sep 08 16:33:50.351240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.351094    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" event={"ID":"606ef2f4-b8c4-4907-9698-67d78cb0456f","Type":"ContainerDied","Data":"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"}
Sep 08 16:33:50.351240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.351145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2" event={"ID":"606ef2f4-b8c4-4907-9698-67d78cb0456f","Type":"ContainerDied","Data":"545d4efef16071876056e25643e7ac4521415e4f8ac88eca85d4923ca99a1830"}
Sep 08 16:33:50.351240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.351166    2571 scope.go:117] "RemoveContainer" containerID="fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"
Sep 08 16:33:50.361514 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.361495    2571 scope.go:117] "RemoveContainer" containerID="fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"
Sep 08 16:33:50.361817 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:33:50.361797    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a\": container with ID starting with fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a not found: ID does not exist" containerID="fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"
Sep 08 16:33:50.361880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.361827    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a"} err="failed to get container status \"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a\": rpc error: code = NotFound desc = could not find container \"fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a\": container with ID starting with fd240280b2811b99bd34386118cfe778b687c0d61c5bcd5cc2bb52c93bf86d0a not found: ID does not exist"
Sep 08 16:33:50.389209 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.389144    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:33:50.401083 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.401055    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b41034-54b85bf5f7-557h2"]
Sep 08 16:33:50.594788 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.594744    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:33:50.598371 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:33:50.598346    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f0c6ae_3e9c_4fca_b938_1a18a343640a.slice/crio-fb0541b3829bf54ac3fa8288603c463015464d44a950b9c38b1f434cb926d378 WatchSource:0}: Error finding container fb0541b3829bf54ac3fa8288603c463015464d44a950b9c38b1f434cb926d378: Status 404 returned error can't find the container with id fb0541b3829bf54ac3fa8288603c463015464d44a950b9c38b1f434cb926d378
Sep 08 16:33:50.599757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.599739    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:33:50.887672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:50.887633    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="606ef2f4-b8c4-4907-9698-67d78cb0456f" path="/var/lib/kubelet/pods/606ef2f4-b8c4-4907-9698-67d78cb0456f/volumes"
Sep 08 16:33:51.357306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:51.357278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" event={"ID":"12f0c6ae-3e9c-4fca-b938-1a18a343640a","Type":"ContainerStarted","Data":"fb0541b3829bf54ac3fa8288603c463015464d44a950b9c38b1f434cb926d378"}
Sep 08 16:33:52.364138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:52.364097    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" event={"ID":"12f0c6ae-3e9c-4fca-b938-1a18a343640a","Type":"ContainerStarted","Data":"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"}
Sep 08 16:33:52.364594 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:52.364299    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:52.382958 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:52.382915    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" podStartSLOduration=3.632731287 podStartE2EDuration="4.382900939s" totalImagesPullingTime="750.169652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:52.381654706 +0000 UTC m=+1262.025731400" watchObservedRunningTime="2026-09-08 16:33:52.382900939 +0000 UTC m=+1262.026977631"
Sep 08 16:33:53.006341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.006308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:53.107515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.107486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:53.204428 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.204392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:53.650638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.650605    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:53.920605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.920533    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:33:53.920934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.920917    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="606ef2f4-b8c4-4907-9698-67d78cb0456f" containerName="maas-api"
Sep 08 16:33:53.920981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.920935    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="606ef2f4-b8c4-4907-9698-67d78cb0456f" containerName="maas-api"
Sep 08 16:33:53.921026 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.920992    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="606ef2f4-b8c4-4907-9698-67d78cb0456f" containerName="maas-api"
Sep 08 16:33:53.923871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.923847    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:53.926946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.926926    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ebb057-openshift-default-dockercfg-nd7qw\""
Sep 08 16:33:53.937583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.937561    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:33:53.950604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:53.950579    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:54.002119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002432    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002476    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.002582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.002506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9s6\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103293    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9s6\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103397    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103534    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103599    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.103927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103712    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.104011 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.103991    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.104078 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.104053    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.104459 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.104441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.105872 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.105855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.106104 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.106084    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.114530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.114506    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9s6\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.115522 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.115503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token\") pod \"e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") " pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.228010 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.227925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:54.233845 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.233822    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:54.537614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.537535    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:33:54.540512 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:33:54.540487    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cde12d_1e6a_403a_87d3_191d90e7e10b.slice/crio-ea4fb8790332e015a97bcae7933770544eb9d4f58787e7b0b549fc8d2d84accc WatchSource:0}: Error finding container ea4fb8790332e015a97bcae7933770544eb9d4f58787e7b0b549fc8d2d84accc: Status 404 returned error can't find the container with id ea4fb8790332e015a97bcae7933770544eb9d4f58787e7b0b549fc8d2d84accc
Sep 08 16:33:54.542727 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.542699    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:54.542802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.542757    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:54.542846 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:54.542802    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:55.396252 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:55.396213    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerStarted","Data":"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"}
Sep 08 16:33:55.396743 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:55.396272    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerStarted","Data":"ea4fb8790332e015a97bcae7933770544eb9d4f58787e7b0b549fc8d2d84accc"}
Sep 08 16:33:55.427930 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:55.427876    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podStartSLOduration=2.427860195 podStartE2EDuration="2.427860195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:55.426309183 +0000 UTC m=+1265.070385886" watchObservedRunningTime="2026-09-08 16:33:55.427860195 +0000 UTC m=+1265.071936887"
Sep 08 16:33:56.234729 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.234685    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:56.239678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.239652    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:56.401222 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.401191    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:56.402141 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.402122    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:33:56.622537 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.622504    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:56.723641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.723604    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:56.882044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.881944    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:33:56.883160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.882557    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy" containerID="cri-o://15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8" gracePeriod=30
Sep 08 16:33:56.905757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:56.905730    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.320052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.320023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.413888 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.413855    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:33:57.417047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.417011    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.507402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.507366    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.711436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.711351    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:33:57.715231 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.715207    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.717790 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.717746    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bce20b-openshift-default-dockercfg-z72fx\""
Sep 08 16:33:57.731908 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.731885    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:33:57.782676 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.782639    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.838536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lc2t\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.838919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.838867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.893452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.893364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:57.939577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lc2t\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.939803 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939600    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.939803 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.939803 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939889    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.939922    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.940177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.940292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940329 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.940295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.940327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.940406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.940398    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.942310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.942289    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.942455 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.942438    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.953447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.953413    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:57.954592 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:57.954571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lc2t\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t\") pod \"e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") " pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:58.026116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.026029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:33:58.325760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.325735    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:33:58.327848 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:33:58.327821    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09999f74_5a99_4c6e_9d21_a3e7f0070ce5.slice/crio-80b122b9dfd16cb06b68c27677a05b64215b04cb802d11242b807191cc1cdb72 WatchSource:0}: Error finding container 80b122b9dfd16cb06b68c27677a05b64215b04cb802d11242b807191cc1cdb72: Status 404 returned error can't find the container with id 80b122b9dfd16cb06b68c27677a05b64215b04cb802d11242b807191cc1cdb72
Sep 08 16:33:58.330177 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.330146    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:58.330266 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.330210    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:58.330266 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.330238    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:33:58.391027 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.390999    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:33:58.415257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.415228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerStarted","Data":"80b122b9dfd16cb06b68c27677a05b64215b04cb802d11242b807191cc1cdb72"}
Sep 08 16:33:58.529705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.529669    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:58.715251 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.715162    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:58.817505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:58.817472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:59.070685 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:59.070649    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:59.174145 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:59.174110    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:33:59.423890 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:59.423801    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerStarted","Data":"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"}
Sep 08 16:33:59.447959 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:33:59.447907    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podStartSLOduration=2.447890203 podStartE2EDuration="2.447890203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:59.446142129 +0000 UTC m=+1269.090218838" watchObservedRunningTime="2026-09-08 16:33:59.447890203 +0000 UTC m=+1269.091966894"
Sep 08 16:34:00.026374 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:00.026335    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:00.032001 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:00.031975    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:00.430343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:00.430312    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:00.433505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:00.433479    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:01.328998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:01.328960    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:01.422374 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:01.422345    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:01.593998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:01.593162    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:49908->10.134.0.41:15021: read: connection reset by peer" start-of-body=
Sep 08 16:34:01.593998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:01.593234    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:49908->10.134.0.41:15021: read: connection reset by peer"
Sep 08 16:34:02.144081 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.144057    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:34:02.286107 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286027    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286107 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286070    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286107 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286097    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286114    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286154    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qdjl\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286181    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286224    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286308    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket\") pod \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\" (UID: \"d674f56f-ae3c-4b9a-bfae-f0916ba454e5\") "
Sep 08 16:34:02.286388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286351    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:02.286733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286543    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:02.286733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286566    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:02.286733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:02.286733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286609    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.286932 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.286747    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:02.288411 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.288390    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:02.288550 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.288530    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl" (OuterVolumeSpecName: "kube-api-access-4qdjl") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "kube-api-access-4qdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:02.288991 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.288966    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:02.288991 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.288968    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d674f56f-ae3c-4b9a-bfae-f0916ba454e5" (UID: "d674f56f-ae3c-4b9a-bfae-f0916ba454e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:34:02.387663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387639    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387663    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387672    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387681    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387691    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387699    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387706    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.387819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.387714    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qdjl\" (UniqueName: \"kubernetes.io/projected/d674f56f-ae3c-4b9a-bfae-f0916ba454e5-kube-api-access-4qdjl\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:02.437086 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.437058    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:34:02.437472 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.437457    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:02.437472 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.437473    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:02.437646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.437538    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:02.437646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.437553    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:02.441073 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.441026    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.443006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.442984    2571 generic.go:356] "Generic (PLEG): container finished" podID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerID="c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe" exitCode=0
Sep 08 16:34:02.443098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.443058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerDied","Data":"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"}
Sep 08 16:34:02.443528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.443511    2571 scope.go:117] "RemoveContainer" containerID="c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"
Sep 08 16:34:02.444798 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.444598    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-bce20b-serving-cert\""
Sep 08 16:34:02.444898 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.444763    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:02.444898 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.444864    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:02.445016 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.444908    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:02.445436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.445206    2571 generic.go:356] "Generic (PLEG): container finished" podID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerID="15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8" exitCode=0
Sep 08 16:34:02.445436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.445243    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerDied","Data":"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"}
Sep 08 16:34:02.445436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.445269    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"
Sep 08 16:34:02.445436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.445291    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm" event={"ID":"d674f56f-ae3c-4b9a-bfae-f0916ba454e5","Type":"ContainerDied","Data":"73e102c54c9e58beae44bb02afe7a676b59c71121a4c371e2fcf61ffd61054a8"}
Sep 08 16:34:02.445436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.445321    2571 scope.go:117] "RemoveContainer" containerID="15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"
Sep 08 16:34:02.454228 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.454009    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:34:02.461600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.461583    2571 scope.go:117] "RemoveContainer" containerID="9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"
Sep 08 16:34:02.470677 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.470644    2571 scope.go:117] "RemoveContainer" containerID="15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"
Sep 08 16:34:02.471057 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:02.471030    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8\": container with ID starting with 15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8 not found: ID does not exist" containerID="15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"
Sep 08 16:34:02.471423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.471069    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8"} err="failed to get container status \"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8\": rpc error: code = NotFound desc = could not find container \"15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8\": container with ID starting with 15471ca8288bc76427cb8c1216de107cb761e5f5107846dad2f81b704a0748e8 not found: ID does not exist"
Sep 08 16:34:02.471423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.471125    2571 scope.go:117] "RemoveContainer" containerID="9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"
Sep 08 16:34:02.471860 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:02.471642    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a\": container with ID starting with 9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a not found: ID does not exist" containerID="9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"
Sep 08 16:34:02.471860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.471677    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a"} err="failed to get container status \"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a\": rpc error: code = NotFound desc = could not find container \"9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a\": container with ID starting with 9d11a12e78ad277ed9292ec688b66d85137435a1a9e9b6e96a782dae4802e72a not found: ID does not exist"
Sep 08 16:34:02.478252 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.478227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:02.488689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.488563    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:34:02.490413 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.490395    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b41034-openshift-default-84c4d7bdd8-z44lm"]
Sep 08 16:34:02.580808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.580750    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:02.589887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.589857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.590040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.589898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88hb\" (UniqueName: \"kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.590040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.589935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.590393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.590187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.691634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.691598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.692115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.691654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88hb\" (UniqueName: \"kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.692115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.691692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.692115 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:02.691782    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-serving-cert: secret "maas-api-e2e-shared-a-w3-bce20b-serving-cert" not found
Sep 08 16:34:02.692115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.691805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.692115 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:02.691858    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls podName:4242e669-9d37-48db-88cc-aa30436f84b8 nodeName:}" failed. No retries permitted until 2026-09-08 16:34:03.19183588 +0000 UTC m=+1272.835912551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls") pod "maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" (UID: "4242e669-9d37-48db-88cc-aa30436f84b8") : secret "maas-api-e2e-shared-a-w3-bce20b-serving-cert" not found
Sep 08 16:34:02.692432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.692412    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.694630 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.694605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.700476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.700454    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88hb\" (UniqueName: \"kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:02.789453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.789415    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:02.888853 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.888752    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" path="/var/lib/kubelet/pods/d674f56f-ae3c-4b9a-bfae-f0916ba454e5/volumes"
Sep 08 16:34:02.985067 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:02.985031    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:03.078996 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.078964    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:03.111279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.111251    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:34:03.111633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.111620    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:03.111682 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.111634    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674f56f-ae3c-4b9a-bfae-f0916ba454e5" containerName="istio-proxy"
Sep 08 16:34:03.115947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.115928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.118283 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.118263    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bce20b\""
Sep 08 16:34:03.118390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.118303    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bce20b-dockercfg-bkxds\""
Sep 08 16:34:03.128528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.128504    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:34:03.197036 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.196976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mz5\" (UniqueName: \"kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.197036 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.197010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.197197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.197102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:03.199587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.199566    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:03.298540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.298491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mz5\" (UniqueName: \"kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.298540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.298542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.299330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.299302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.306973 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.306951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mz5\" (UniqueName: \"kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5\") pod \"payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.359147 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.359103    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:03.428110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.428078    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:03.455091 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.455032    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerStarted","Data":"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"}
Sep 08 16:34:03.585816 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.579983    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:03.585816 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.580466    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:03.585816 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.580539    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:03.657921 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.657898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:34:03.659848 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:34:03.659822    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4242e669_9d37_48db_88cc_aa30436f84b8.slice/crio-07140653c9eea385d13a76396dc3f396143ecd19c4ebe753003ef57ceb7a5031 WatchSource:0}: Error finding container 07140653c9eea385d13a76396dc3f396143ecd19c4ebe753003ef57ceb7a5031: Status 404 returned error can't find the container with id 07140653c9eea385d13a76396dc3f396143ecd19c4ebe753003ef57ceb7a5031
Sep 08 16:34:03.734356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:03.734221    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:34:03.736648 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:34:03.736617    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7a8930_c090_499d_9431_9794be45cd63.slice/crio-8eb837da396613667dc33db46c4df82ec4f91a062030f14505081c79389c584b WatchSource:0}: Error finding container 8eb837da396613667dc33db46c4df82ec4f91a062030f14505081c79389c584b: Status 404 returned error can't find the container with id 8eb837da396613667dc33db46c4df82ec4f91a062030f14505081c79389c584b
Sep 08 16:34:04.463600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.462890    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" event={"ID":"4242e669-9d37-48db-88cc-aa30436f84b8","Type":"ContainerStarted","Data":"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"}
Sep 08 16:34:04.463600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.462995    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" event={"ID":"4242e669-9d37-48db-88cc-aa30436f84b8","Type":"ContainerStarted","Data":"07140653c9eea385d13a76396dc3f396143ecd19c4ebe753003ef57ceb7a5031"}
Sep 08 16:34:04.463600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.463231    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:04.466657 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.466508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" event={"ID":"7d7a8930-c090-499d-9431-9794be45cd63","Type":"ContainerStarted","Data":"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"}
Sep 08 16:34:04.466657 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.466559    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:04.466657 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.466581    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" event={"ID":"7d7a8930-c090-499d-9431-9794be45cd63","Type":"ContainerStarted","Data":"8eb837da396613667dc33db46c4df82ec4f91a062030f14505081c79389c584b"}
Sep 08 16:34:04.488851 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.488304    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" podStartSLOduration=1.938638016 podStartE2EDuration="2.488282153s" totalImagesPullingTime="549.644137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:04.486140919 +0000 UTC m=+1274.130217615" watchObservedRunningTime="2026-09-08 16:34:04.488282153 +0000 UTC m=+1274.132358846"
Sep 08 16:34:04.503685 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.503623    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" podStartSLOduration=1.503600911 podStartE2EDuration="1.503600911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:04.501837453 +0000 UTC m=+1274.145914149" watchObservedRunningTime="2026-09-08 16:34:04.503600911 +0000 UTC m=+1274.147677604"
Sep 08 16:34:04.574524 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.574476    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:04.574733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:04.574550    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:05.574541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:05.574097    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:05.574541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:05.574159    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:06.574188 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:06.574148    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:06.574479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:06.574446    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:06.597381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:06.597347    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:34:07.574490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:07.574072    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:07.574490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:07.574134    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:08.573976 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:08.573935    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 16:34:08.574587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:08.574539    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 16:34:09.583190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:09.583164    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:09.584666 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:09.584606    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:09.586393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:09.586371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:09.734013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:09.733976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:10.497957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:10.497907    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:34:12.196397 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:12.196364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:12.442940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:12.442901    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 16:34:12.443105 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:12.442968    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 16:34:12.839750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:12.839708    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:12.939237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:12.939204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:13.261229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.261089    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:13.521707 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.521622    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerID="6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230" exitCode=0
Sep 08 16:34:13.521904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.521701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerDied","Data":"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"}
Sep 08 16:34:13.522101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.522087    2571 scope.go:117] "RemoveContainer" containerID="6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"
Sep 08 16:34:13.523155 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.523129    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:13.523233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.523218    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:13.523273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:13.523267    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:14.530745 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:14.530699    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerStarted","Data":"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"}
Sep 08 16:34:15.234327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:15.234287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:15.234509 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:15.234487    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 08 16:34:15.234595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:15.234572    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 08 16:34:15.476058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:15.476023    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:34:16.235247 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:16.235205    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 08 16:34:16.235634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:16.235265    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 08 16:34:16.473282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:16.472927    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 16:34:16.473282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:16.472995    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 16:34:17.234598 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.234562    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 08 16:34:17.235520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.234617    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 08 16:34:17.549544 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.549449    2571 generic.go:356] "Generic (PLEG): container finished" podID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerID="c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251" exitCode=0
Sep 08 16:34:17.549544 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.549505    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerDied","Data":"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"}
Sep 08 16:34:17.549925 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.549907    2571 scope.go:117] "RemoveContainer" containerID="c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"
Sep 08 16:34:17.551061 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.551030    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:17.551125 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.551114    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:17.551165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:17.551156    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:18.235121 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:18.235083    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 08 16:34:18.235302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:18.235139    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 08 16:34:18.557335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:18.557245    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerStarted","Data":"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"}
Sep 08 16:34:19.027176 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.027141    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:19.027525 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.027493    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 08 16:34:19.027633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.027567    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 08 16:34:19.151423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.151379    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:19.229048 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.229018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:19.234896 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.234872    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 08 16:34:19.234989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.234924    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 08 16:34:19.525410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:19.525361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:20.027390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.027354    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 08 16:34:20.027758 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.027407    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 08 16:34:20.273567 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.273533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:20.295138 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.295050    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:20.295362 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.295344    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:20.296123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:20.296105    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:21.027100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:21.027057    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 08 16:34:21.027271 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:21.027123    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 08 16:34:21.253961 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:21.253925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:22.026573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:22.026536    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 08 16:34:22.026787 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:22.026593    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 08 16:34:22.229496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:22.229450    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:23.026405 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:23.026367    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 08 16:34:23.026820 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:23.026421    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 08 16:34:23.418990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:23.418949    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:24.029798 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:24.029735    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:24.030279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:24.030137    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:24.030786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:24.030751    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:34:26.656417 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:26.656380    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:26.759167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:26.759124    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:26.893288 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:26.893241    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:26.997733 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:26.997620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:27.605550 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:27.605508    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:29.673323 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:29.673293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.299611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.299564    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.332202 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.332166    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:34:30.332433 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.332409    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" podUID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" containerName="maas-api" containerID="cri-o://d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e" gracePeriod=30
Sep 08 16:34:30.402492 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.402449    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.610039 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.610018    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:34:30.635241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.635212    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.640447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.640424    2571 generic.go:356] "Generic (PLEG): container finished" podID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" containerID="d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e" exitCode=0
Sep 08 16:34:30.640535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.640476    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"
Sep 08 16:34:30.640535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.640509    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" event={"ID":"12f0c6ae-3e9c-4fca-b938-1a18a343640a","Type":"ContainerDied","Data":"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"}
Sep 08 16:34:30.640626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.640558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml" event={"ID":"12f0c6ae-3e9c-4fca-b938-1a18a343640a","Type":"ContainerDied","Data":"fb0541b3829bf54ac3fa8288603c463015464d44a950b9c38b1f434cb926d378"}
Sep 08 16:34:30.640626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.640583    2571 scope.go:117] "RemoveContainer" containerID="d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"
Sep 08 16:34:30.649758 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.649707    2571 scope.go:117] "RemoveContainer" containerID="d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"
Sep 08 16:34:30.650083 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:30.650057    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e\": container with ID starting with d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e not found: ID does not exist" containerID="d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"
Sep 08 16:34:30.650185 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.650091    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e"} err="failed to get container status \"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e\": rpc error: code = NotFound desc = could not find container \"d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e\": container with ID starting with d42544c328b5a42e5cad72ef9465651e2d5aba8b9e1d373c0525bee2b3e9293e not found: ID does not exist"
Sep 08 16:34:30.687654 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.687628    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") pod \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") "
Sep 08 16:34:30.688098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.687674    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs\") pod \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") "
Sep 08 16:34:30.688098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.687708    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfgvt\" (UniqueName: \"kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt\") pod \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") "
Sep 08 16:34:30.688098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.687850    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca\") pod \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\" (UID: \"12f0c6ae-3e9c-4fca-b938-1a18a343640a\") "
Sep 08 16:34:30.688279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.688231    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12f0c6ae-3e9c-4fca-b938-1a18a343640a" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:30.689937 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.689915    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12f0c6ae-3e9c-4fca-b938-1a18a343640a" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:34:30.690025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.689973    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12f0c6ae-3e9c-4fca-b938-1a18a343640a" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:34:30.690065 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.690050    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt" (OuterVolumeSpecName: "kube-api-access-cfgvt") pod "12f0c6ae-3e9c-4fca-b938-1a18a343640a" (UID: "12f0c6ae-3e9c-4fca-b938-1a18a343640a"). InnerVolumeSpecName "kube-api-access-cfgvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:30.764238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.764206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.789314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.789295    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f0c6ae-3e9c-4fca-b938-1a18a343640a-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.789393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.789316    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.789393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.789326    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f0c6ae-3e9c-4fca-b938-1a18a343640a-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.789393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.789334    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgvt\" (UniqueName: \"kubernetes.io/projected/12f0c6ae-3e9c-4fca-b938-1a18a343640a-kube-api-access-cfgvt\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.949995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.949939    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:30.963197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.963172    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:34:30.971235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:30.971210    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-57c06fce-5b8c8c467c-r4rml"]
Sep 08 16:34:31.112177 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:31.112144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:31.313786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:31.313655    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:31.449825 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:31.449758    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:31.599723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:31.599690    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:34:32.462967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.462931    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:34:32.463390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.463377    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" containerName="maas-api"
Sep 08 16:34:32.463438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.463393    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" containerName="maas-api"
Sep 08 16:34:32.463473 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.463457    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" containerName="maas-api"
Sep 08 16:34:32.467909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.467891    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.473309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.473288    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e22de8-serving-cert\""
Sep 08 16:34:32.480346 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.480322    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:34:32.506913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.506888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.507025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.506949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.507025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.507021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwg6\" (UniqueName: \"kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.507150 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.507127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.524402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.524376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:32.607968 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.607928    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.607968 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.607971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.608237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.608015    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.608237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.608051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwg6\" (UniqueName: \"kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.608237 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:32.608171    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-serving-cert: secret "maas-api-e2e-shared-b-w3-e22de8-serving-cert" not found
Sep 08 16:34:32.608237 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:32.608238    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls podName:8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f nodeName:}" failed. No retries permitted until 2026-09-08 16:34:33.108220545 +0000 UTC m=+1302.752297214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" (UID: "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f") : secret "maas-api-e2e-shared-b-w3-e22de8-serving-cert" not found
Sep 08 16:34:32.608725 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.608707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.610633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.610612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.620477 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.620457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwg6\" (UniqueName: \"kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:32.622864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.622839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:32.827614 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.827587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:32.887624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:32.887597    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f0c6ae-3e9c-4fca-b938-1a18a343640a" path="/var/lib/kubelet/pods/12f0c6ae-3e9c-4fca-b938-1a18a343640a/volumes"
Sep 08 16:34:33.112955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.112870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:33.115416 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.115387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:33.223912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.223879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:33.343639 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.343601    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:33.379628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.379561    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:33.655909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:33.655885    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:34:33.658448 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:34:33.658421    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdd3173_dc7f_4b31_8487_2eb13f2d5c7f.slice/crio-3e25eb80ef3c5d37efbb15bbc14a3898337450de41fc4188e931323faa74409f WatchSource:0}: Error finding container 3e25eb80ef3c5d37efbb15bbc14a3898337450de41fc4188e931323faa74409f: Status 404 returned error can't find the container with id 3e25eb80ef3c5d37efbb15bbc14a3898337450de41fc4188e931323faa74409f
Sep 08 16:34:34.664382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:34.664343    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" event={"ID":"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f","Type":"ContainerStarted","Data":"8ef8f8f16755418460d54043349e91704c10329f83c00cfd2f2d65a96a6a34c7"}
Sep 08 16:34:34.664800 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:34.664408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" event={"ID":"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f","Type":"ContainerStarted","Data":"3e25eb80ef3c5d37efbb15bbc14a3898337450de41fc4188e931323faa74409f"}
Sep 08 16:34:34.664800 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:34.664438    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:34.682753 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:34.682704    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" podStartSLOduration=2.319307658 podStartE2EDuration="2.682688898s" totalImagesPullingTime="363.38124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:34.681429639 +0000 UTC m=+1304.325506331" watchObservedRunningTime="2026-09-08 16:34:34.682688898 +0000 UTC m=+1304.326765581"
Sep 08 16:34:35.257014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:35.256968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:37.491497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:37.491446    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:37.600855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:37.600818    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:37.719964 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:37.719933    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:34:37.720186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:37.720157    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" containerID="cri-o://7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e" gracePeriod=30
Sep 08 16:34:37.734014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:37.733978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:38.118943 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:38.118906    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:38.219702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:38.219666    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:38.327532 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:38.327484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:39.586356 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:39.586317    2571 patch_prober.go:28] interesting pod/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 16:34:39.586721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:39.586392    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:34:40.675175 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:40.675142    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:34:40.785533 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:40.785496    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:41.525932 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:41.525896    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:41.627003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:41.626970    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:41.727576 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:41.727543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:41.826098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:41.826066    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:41.900216 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:41.900182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:42.172474 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:42.172390    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:42.973217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:42.973193    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:43.102799 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102753    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.102944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102804    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.102944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102855    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.102944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102895    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.102944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102933    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.103146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.102970    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.103146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103013    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctv95\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.103146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103057    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.103146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103050    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:43.103146 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103100    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo\") pod \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\" (UID: \"2d68f6a7-31f3-4443-9584-c35231c9c9d6\") "
Sep 08 16:34:43.103410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103227    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:43.103410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103245    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:43.103410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103299    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:43.103410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103285    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:43.103623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103454    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.103623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103473    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.103623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103492    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.103623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103510    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.103623 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.103525    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.105307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.105282    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95" (OuterVolumeSpecName: "kube-api-access-ctv95") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "kube-api-access-ctv95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:43.105307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.105305    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:43.105470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.105370    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:43.105470 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.105412    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d68f6a7-31f3-4443-9584-c35231c9c9d6" (UID: "2d68f6a7-31f3-4443-9584-c35231c9c9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:34:43.204897 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.204865    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv95\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-kube-api-access-ctv95\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.204897 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.204892    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.205046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.204905    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.205046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.204916    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d68f6a7-31f3-4443-9584-c35231c9c9d6-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:43.710957 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.710923    2571 generic.go:356] "Generic (PLEG): container finished" podID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerID="7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e" exitCode=0
Sep 08 16:34:43.711120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.710972    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"
Sep 08 16:34:43.711120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.710988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerDied","Data":"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"}
Sep 08 16:34:43.711120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.711030    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4" event={"ID":"2d68f6a7-31f3-4443-9584-c35231c9c9d6","Type":"ContainerDied","Data":"71104f3a91255d41840af26b1460b1b53eee7a05bbf017253fe2c77411a81991"}
Sep 08 16:34:43.711120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.711053    2571 scope.go:117] "RemoveContainer" containerID="7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"
Sep 08 16:34:43.720588 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.720565    2571 scope.go:117] "RemoveContainer" containerID="c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"
Sep 08 16:34:43.728281 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.728266    2571 scope.go:117] "RemoveContainer" containerID="7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"
Sep 08 16:34:43.728537 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:43.728516    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e\": container with ID starting with 7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e not found: ID does not exist" containerID="7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"
Sep 08 16:34:43.728611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.728548    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e"} err="failed to get container status \"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e\": rpc error: code = NotFound desc = could not find container \"7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e\": container with ID starting with 7bbbb6d8045682ede0e189a0f148f0ab94dbde902b6a60d24b9e272d368f6e1e not found: ID does not exist"
Sep 08 16:34:43.728611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.728571    2571 scope.go:117] "RemoveContainer" containerID="c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"
Sep 08 16:34:43.728841 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:43.728822    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe\": container with ID starting with c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe not found: ID does not exist" containerID="c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"
Sep 08 16:34:43.728899 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.728847    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe"} err="failed to get container status \"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe\": rpc error: code = NotFound desc = could not find container \"c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe\": container with ID starting with c00d9eaa8b52ebd7dacdf0118541f06330433baadedc8ac713abb365aa14abbe not found: ID does not exist"
Sep 08 16:34:43.733653 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.733634    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:34:43.737948 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:43.737929    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-57c06fce-openshift-default-d6b79676d-k6th4"]
Sep 08 16:34:44.176528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:44.176490    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:34:44.553160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:44.553068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:44.665374 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:44.665337    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:44.766546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:44.766515    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:44.888200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:44.888172    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" path="/var/lib/kubelet/pods/2d68f6a7-31f3-4443-9584-c35231c9c9d6/volumes"
Sep 08 16:34:45.046634 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:45.046594    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:49.427990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:49.427954    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:49.529183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:49.529149    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:49.743793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:49.743695    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:34:49.743950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:49.743929    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" containerID="cri-o://646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b" gracePeriod=30
Sep 08 16:34:49.758465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:49.758433    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:50.297496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:50.297460    2571 patch_prober.go:28] interesting pod/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 16:34:50.297669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:50.297524    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:34:50.460006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:50.459972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:50.630436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:50.630405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:51.426893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:51.426850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:52.089672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:52.089640    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:52.279167 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:52.279136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:52.431396 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:52.431304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:53.071351 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:53.071311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:53.298339 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:53.298307    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:53.727375 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:53.727335    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:53.958330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:53.958296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:54.264310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264282    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:34:54.264646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264634    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:54.264705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264648    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:54.264705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264662    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:54.264705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264668    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:54.264829 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.264717    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:54.268948 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.268931    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.272153 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.272134    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-8ac343-openshift-default-dockercfg-247ns\""
Sep 08 16:34:54.283057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.283037    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:34:54.323493 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.323467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:54.400449 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400637    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwg4w\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400707    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.400858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400802    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.401024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.400860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501186    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:54.501365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwg4w\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501529    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.501989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501963    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.502101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.501993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.502156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.502137    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.503951 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.503923    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.504168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.504150    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.514147 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.514123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwg4w\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.514276 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.514253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token\") pod \"e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") " pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.579123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.579101    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:54.856909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.856884    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:34:54.859486 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:34:54.859458    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b0d3aa_33a2_4061_938c_1ad06acb8801.slice/crio-6a2ce1283cc9a776f66161454a6e2c73cc45b67f46402d4c30f2e8ffa33baeea WatchSource:0}: Error finding container 6a2ce1283cc9a776f66161454a6e2c73cc45b67f46402d4c30f2e8ffa33baeea: Status 404 returned error can't find the container with id 6a2ce1283cc9a776f66161454a6e2c73cc45b67f46402d4c30f2e8ffa33baeea
Sep 08 16:34:54.862182 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.862151    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:54.862257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.862216    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:54.862257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.862244    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:34:54.988008 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:54.987984    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:55.107975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.107889    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.107975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.107957    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.107987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108005    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108034    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108057    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s9s6\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108083    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108115    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108167    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token\") pod \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\" (UID: \"f8cde12d-1e6a-403a-87d3-191d90e7e10b\") "
Sep 08 16:34:55.108564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108400    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:55.108564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108460    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:55.108679 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108575    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.108679 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108595    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.108948 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108924    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:55.109064 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.108998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:55.109226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.109200    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:55.110596 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.110577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:34:55.111094 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.111066    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:55.111094 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.111084    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:34:55.111221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.111113    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6" (OuterVolumeSpecName: "kube-api-access-7s9s6") pod "f8cde12d-1e6a-403a-87d3-191d90e7e10b" (UID: "f8cde12d-1e6a-403a-87d3-191d90e7e10b"). InnerVolumeSpecName "kube-api-access-7s9s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:55.209547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209512    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209542    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209552    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209560    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9s6\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-kube-api-access-7s9s6\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209571    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cde12d-1e6a-403a-87d3-191d90e7e10b-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209579    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.209802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.209586    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cde12d-1e6a-403a-87d3-191d90e7e10b-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:34:55.338905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.338869    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:55.774629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.774552    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerID="646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b" exitCode=0
Sep 08 16:34:55.774629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.774615    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"
Sep 08 16:34:55.774903 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.774619    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerDied","Data":"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"}
Sep 08 16:34:55.774903 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.774809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6" event={"ID":"f8cde12d-1e6a-403a-87d3-191d90e7e10b","Type":"ContainerDied","Data":"ea4fb8790332e015a97bcae7933770544eb9d4f58787e7b0b549fc8d2d84accc"}
Sep 08 16:34:55.774903 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.774843    2571 scope.go:117] "RemoveContainer" containerID="646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"
Sep 08 16:34:55.776551 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.776528    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerStarted","Data":"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"}
Sep 08 16:34:55.776641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.776586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerStarted","Data":"6a2ce1283cc9a776f66161454a6e2c73cc45b67f46402d4c30f2e8ffa33baeea"}
Sep 08 16:34:55.784785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.784751    2571 scope.go:117] "RemoveContainer" containerID="6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"
Sep 08 16:34:55.794864 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.793326    2571 scope.go:117] "RemoveContainer" containerID="646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"
Sep 08 16:34:55.795244 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:55.795225    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b\": container with ID starting with 646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b not found: ID does not exist" containerID="646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"
Sep 08 16:34:55.795307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.795253    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b"} err="failed to get container status \"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b\": rpc error: code = NotFound desc = could not find container \"646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b\": container with ID starting with 646b2ef9a6c573dcd168f31a32ea13992e158e4b97016862ed1b2d8d49ba979b not found: ID does not exist"
Sep 08 16:34:55.795307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.795269    2571 scope.go:117] "RemoveContainer" containerID="6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"
Sep 08 16:34:55.795559 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:55.795542    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230\": container with ID starting with 6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230 not found: ID does not exist" containerID="6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"
Sep 08 16:34:55.795607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.795566    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230"} err="failed to get container status \"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230\": rpc error: code = NotFound desc = could not find container \"6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230\": container with ID starting with 6407891196a975876150e3a89f696443737fc82573846cba51d26ef319196230 not found: ID does not exist"
Sep 08 16:34:55.808343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.808302    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podStartSLOduration=1.808290122 podStartE2EDuration="1.808290122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:55.805235017 +0000 UTC m=+1325.449311710" watchObservedRunningTime="2026-09-08 16:34:55.808290122 +0000 UTC m=+1325.452366814"
Sep 08 16:34:55.831334 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.831312    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:34:55.841426 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:55.841402    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ebb057-openshift-default-777bdcb984-8xbk6"]
Sep 08 16:34:56.579258 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:56.579228    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:56.584254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:56.584231    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:56.783096 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:56.783070    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:56.783885 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:56.783866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:34:56.888579 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:56.888506    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" path="/var/lib/kubelet/pods/f8cde12d-1e6a-403a-87d3-191d90e7e10b/volumes"
Sep 08 16:34:57.879540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:57.879510    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:58.012071 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.012035    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:58.978989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.978959    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:34:58.979343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979329    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.979388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979345    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.979388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979360    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.979388 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979371    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.979485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979437    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.979485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979452    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d68f6a7-31f3-4443-9584-c35231c9c9d6" containerName="istio-proxy"
Sep 08 16:34:58.979485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.979458    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cde12d-1e6a-403a-87d3-191d90e7e10b" containerName="istio-proxy"
Sep 08 16:34:58.983927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.983906    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:58.987130 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:58.986935    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8ac343-serving-cert\""
Sep 08 16:34:59.092078 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.092050    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:34:59.147142 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.147105    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:59.150181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.150153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ps8\" (UniqueName: \"kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.150302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.150193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.150302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.150250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.150302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.150268    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.251840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.251745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.251840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.251806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.252070 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.251876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ps8\" (UniqueName: \"kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.252070 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:59.251907    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-serving-cert: secret "maas-api-e2e-shared-a-w2-8ac343-serving-cert" not found
Sep 08 16:34:59.252070 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:34:59.251986    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls podName:ab4af05d-f22a-4453-aa11-c2bbb07026c1 nodeName:}" failed. No retries permitted until 2026-09-08 16:34:59.751964754 +0000 UTC m=+1329.396041441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" (UID: "ab4af05d-f22a-4453-aa11-c2bbb07026c1") : secret "maas-api-e2e-shared-a-w2-8ac343-serving-cert" not found
Sep 08 16:34:59.252070 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.252008    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.252618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.252595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.254328 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.254311    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.273378 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.273359    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ps8\" (UniqueName: \"kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.302646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.302602    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:59.570206 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.570176    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:34:59.758347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.758308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.760993 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.760969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:34:59.895225 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:34:59.895143    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:35:00.212378 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:00.212349    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:35:00.214495 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:35:00.214468    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4af05d_f22a_4453_aa11_c2bbb07026c1.slice/crio-153050ab70e1ab9fba851cc36a03ffee286ce095c625d25a686576cba3299d71 WatchSource:0}: Error finding container 153050ab70e1ab9fba851cc36a03ffee286ce095c625d25a686576cba3299d71: Status 404 returned error can't find the container with id 153050ab70e1ab9fba851cc36a03ffee286ce095c625d25a686576cba3299d71
Sep 08 16:35:00.336730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:00.336686    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:00.803498 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:00.803467    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" event={"ID":"ab4af05d-f22a-4453-aa11-c2bbb07026c1","Type":"ContainerStarted","Data":"153050ab70e1ab9fba851cc36a03ffee286ce095c625d25a686576cba3299d71"}
Sep 08 16:35:01.001617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:01.001577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:01.809219 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:01.809179    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" event={"ID":"ab4af05d-f22a-4453-aa11-c2bbb07026c1","Type":"ContainerStarted","Data":"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"}
Sep 08 16:35:01.809652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:01.809241    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:35:01.851597 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:01.851545    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" podStartSLOduration=3.054638569 podStartE2EDuration="3.85153278s" totalImagesPullingTime="796.894211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:01.849949367 +0000 UTC m=+1331.494026059" watchObservedRunningTime="2026-09-08 16:35:01.85153278 +0000 UTC m=+1331.495609471"
Sep 08 16:35:04.491803 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:04.491753    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:35:07.819716 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:07.819685    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:35:09.315566 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:09.315531    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:10.802036 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:10.802007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:10.954052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:10.954007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:11.148024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:11.147991    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:11.253028 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:11.252989    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:15.879819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.879760    2571 generic.go:356] "Generic (PLEG): container finished" podID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerID="22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370" exitCode=0
Sep 08 16:35:15.880232 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.879838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerDied","Data":"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"}
Sep 08 16:35:15.880293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.880275    2571 scope.go:117] "RemoveContainer" containerID="22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"
Sep 08 16:35:15.881422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.881393    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:35:15.881528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.881469    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:35:15.881528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:15.881502    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:35:16.270868 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:16.270743    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:16.892119 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:16.892083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerStarted","Data":"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"}
Sep 08 16:35:17.579273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:17.579229    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:35:17.579556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:17.579524    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 08 16:35:17.579627 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:17.579587    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 08 16:35:18.579972 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:18.579935    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 08 16:35:18.580445 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:18.579995    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 08 16:35:19.579950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:19.579907    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 08 16:35:19.580152 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:19.579972    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 08 16:35:19.838883 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:19.838790    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:35:20.580310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:20.580271    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 08 16:35:20.581282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:20.580872    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 08 16:35:21.579646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:21.579608    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 08 16:35:21.579971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:21.579938    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 08 16:35:22.584382 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:22.584349    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:35:22.584744 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:22.584621    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:35:22.585369 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:22.585351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:35:22.812429 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:22.812394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:23.124441 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:23.124409    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:23.435453 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:23.435374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:23.939741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:23.939709    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:27.046438 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:27.046403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:27.347139 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:27.347107    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:27.559333 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:27.559281    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:28.090262 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:28.090223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:28.188166 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:28.188136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:28.368358 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:28.368275    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:28.649889 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:28.649797    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:28.748379 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:28.748348    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:33.901972 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:33.901940    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:35:42.426507 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:42.426468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:42.551040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:42.550984    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:47.531244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.531213    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:35:47.535980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.535955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.541972 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.541705    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"default-dockercfg-kphbb\""
Sep 08 16:35:47.543208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.543188    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"auth-test-model-e22de8-kserve-self-signed-certs\""
Sep 08 16:35:47.543529 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.543509    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"kube-root-ca.crt\""
Sep 08 16:35:47.543877 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.543595    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"openshift-service-ca.crt\""
Sep 08 16:35:47.553659 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.553637    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:35:47.594321 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594293    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.594487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.594487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.594487 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcb2n\" (UniqueName: \"kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.594639 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.594639 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.594533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.634573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.634473    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:47.695823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.695981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.695981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.695981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcb2n\" (UniqueName: \"kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.695981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695940    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.695981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.695963    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.696303 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.696279    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.696371 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.696341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.696422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.696339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.698900 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.698873    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.699242 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.699220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.709997 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.709971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcb2n\" (UniqueName: \"kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n\") pod \"auth-test-model-e22de8-kserve-78b8fb859f-pfhbm\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:47.846429 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:47.846394    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:48.141541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:48.141515    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:35:48.143324 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:35:48.143296    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod709ade1a_23b5_427c_a71d_85ae9ce22119.slice/crio-56f6030d8f18751c5c94a60759494e1b69affc89c561eaab145744b5f1f96906 WatchSource:0}: Error finding container 56f6030d8f18751c5c94a60759494e1b69affc89c561eaab145744b5f1f96906: Status 404 returned error can't find the container with id 56f6030d8f18751c5c94a60759494e1b69affc89c561eaab145744b5f1f96906
Sep 08 16:35:49.041635 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:49.041538    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" event={"ID":"709ade1a-23b5-427c-a71d-85ae9ce22119","Type":"ContainerStarted","Data":"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"}
Sep 08 16:35:49.041635 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:49.041586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" event={"ID":"709ade1a-23b5-427c-a71d-85ae9ce22119","Type":"ContainerStarted","Data":"56f6030d8f18751c5c94a60759494e1b69affc89c561eaab145744b5f1f96906"}
Sep 08 16:35:49.067950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:49.067906    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" podStartSLOduration=2.067890671 podStartE2EDuration="2.067890671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:49.064888556 +0000 UTC m=+1378.708965246" watchObservedRunningTime="2026-09-08 16:35:49.067890671 +0000 UTC m=+1378.711967393"
Sep 08 16:35:51.697609 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:51.697571    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:35:51.789931 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:51.789898    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:53.873607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:53.873572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:35:57.847231 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:57.847195    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:57.847647 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:57.847278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:57.859492 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:57.859471    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:35:58.094052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:35:58.094021    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:36:04.679529 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:04.679491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:36:05.911653 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:05.911593    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:06.285763 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:06.285674    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:06.755071 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:06.755029    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:09.119512 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:09.119471    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:09.250755 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:09.250716    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:10.495076 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.495043    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:36:10.499729 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.499711    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.502779 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.502742    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1fa1601c-serving-cert\""
Sep 08 16:36:10.529945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.529920    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:36:10.594465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.594440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.594628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.594512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.594628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.594585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.594731 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.594664    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpcj\" (UniqueName: \"kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.631067 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.631025    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:10.696038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.696004    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.696241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.696046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.696241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.696097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpcj\" (UniqueName: \"kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.696241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.696128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.696727 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.696703    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.699024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.699000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.699085 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.699000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.712278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.712259    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpcj\" (UniqueName: \"kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj\") pod \"maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:10.790414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.790321    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:10.811316 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:10.811292    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:11.117760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:11.117724    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:36:11.129183 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:36:11.129157    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd3314f_e47e_4589_8af1_9773667589de.slice/crio-6cba53455dc232cc4401215b4c78a5f98cb59b4665bb3cd6b37ae472de90bd98 WatchSource:0}: Error finding container 6cba53455dc232cc4401215b4c78a5f98cb59b4665bb3cd6b37ae472de90bd98: Status 404 returned error can't find the container with id 6cba53455dc232cc4401215b4c78a5f98cb59b4665bb3cd6b37ae472de90bd98
Sep 08 16:36:11.152587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:11.152554    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" event={"ID":"9cd3314f-e47e-4589-8af1-9773667589de","Type":"ContainerStarted","Data":"6cba53455dc232cc4401215b4c78a5f98cb59b4665bb3cd6b37ae472de90bd98"}
Sep 08 16:36:11.557618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:11.557577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:12.159101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:12.159065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" event={"ID":"9cd3314f-e47e-4589-8af1-9773667589de","Type":"ContainerStarted","Data":"205c2d2bbd944d18e81b5b9f44ee50ebe83659bfbf074f067679ee85eeddcf98"}
Sep 08 16:36:12.159285 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:12.159187    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:12.224460 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:12.224415    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" podStartSLOduration=1.859423757 podStartE2EDuration="2.224401435s" totalImagesPullingTime="364.977678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:12.218808184 +0000 UTC m=+1401.862884868" watchObservedRunningTime="2026-09-08 16:36:12.224401435 +0000 UTC m=+1401.868478126"
Sep 08 16:36:12.736166 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:12.736131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:18.169353 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:18.169321    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:36:20.784243 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:20.784206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:36:23.315134 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:23.315103    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:34.043322 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:34.043290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:36:37.185187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:37.185144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:36:50.798981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:50.798949    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:36:50.966099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:36:50.966067    2571 scope.go:117] "RemoveContainer" containerID="e32897a84221f25e17e5ae5620a4c0006d3a0969737ca605daec5b4558675768"
Sep 08 16:37:03.044257 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:03.044214    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:05.176587 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:05.176553    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:37:13.780734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.780698    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:37:13.785757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.785735    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.789224 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.789187    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1fa1601c\"/\"kube-root-ca.crt\""
Sep 08 16:37:13.789224 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.789186    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1fa1601c\"/\"e2e-lifecycle-model-1fa1601c-kserve-self-signed-certs\""
Sep 08 16:37:13.789430 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.789242    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1fa1601c\"/\"openshift-service-ca.crt\""
Sep 08 16:37:13.789430 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.789257    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1fa1601c\"/\"default-dockercfg-xhsgk\""
Sep 08 16:37:13.792678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.792656    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:37:13.880152 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.880120    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:13.893381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.893547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.893547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.893547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.893711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.893711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.893592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l5z\" (UniqueName: \"kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994659 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994836 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.994999 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.994862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l5z\" (UniqueName: \"kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.995106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.995080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.995158 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.995121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.995203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.995176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.997395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.997373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:13.997500 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:13.997479    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:14.002757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:14.002733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l5z\" (UniqueName: \"kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z\") pod \"e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") " pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:14.095804 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:14.095751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:14.407815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:14.407790    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:37:14.410362 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:37:14.410319    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94f22a52_3b9c_4bcc_b31f_720e4180487c.slice/crio-2f9d6cbc7fbb44d5b17a24e211559478abdc07fbe796a4e8027dab3a3a4b981e WatchSource:0}: Error finding container 2f9d6cbc7fbb44d5b17a24e211559478abdc07fbe796a4e8027dab3a3a4b981e: Status 404 returned error can't find the container with id 2f9d6cbc7fbb44d5b17a24e211559478abdc07fbe796a4e8027dab3a3a4b981e
Sep 08 16:37:14.470717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:14.470684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" event={"ID":"94f22a52-3b9c-4bcc-b31f-720e4180487c","Type":"ContainerStarted","Data":"2f9d6cbc7fbb44d5b17a24e211559478abdc07fbe796a4e8027dab3a3a4b981e"}
Sep 08 16:37:15.478627 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:15.478590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" event={"ID":"94f22a52-3b9c-4bcc-b31f-720e4180487c","Type":"ContainerStarted","Data":"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"}
Sep 08 16:37:15.496882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:15.496835    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" podStartSLOduration=2.496820082 podStartE2EDuration="2.496820082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:15.494521526 +0000 UTC m=+1465.138598218" watchObservedRunningTime="2026-09-08 16:37:15.496820082 +0000 UTC m=+1465.140896775"
Sep 08 16:37:20.479042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:20.479005    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:37:24.095998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:24.095961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:24.096379 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:24.096029    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:24.108011 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:24.107988    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:24.540041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:24.539957    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:37:37.977620 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:37.977584    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:37:44.152082 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:44.152017    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:44.320077 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:44.320044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:45.389454 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:45.389412    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:37:45.413154 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:45.413123    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:50.788042 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:50.788012    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:37:50.907494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:50.907469    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:37:50.911071 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:50.911049    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:37:50.985599 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:50.985565    2571 scope.go:117] "RemoveContainer" containerID="83d8520d381ccd02e316dcaefaaecd099485551067117ade3af5e428a38e800e"
Sep 08 16:37:59.573983 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.573951    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:37:59.574501 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.574198    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" podUID="9cd3314f-e47e-4589-8af1-9773667589de" containerName="maas-api" containerID="cri-o://205c2d2bbd944d18e81b5b9f44ee50ebe83659bfbf074f067679ee85eeddcf98" gracePeriod=30
Sep 08 16:37:59.621695 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.621654    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:59.696792 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.696742    2571 generic.go:356] "Generic (PLEG): container finished" podID="9cd3314f-e47e-4589-8af1-9773667589de" containerID="205c2d2bbd944d18e81b5b9f44ee50ebe83659bfbf074f067679ee85eeddcf98" exitCode=0
Sep 08 16:37:59.696911 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.696805    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" event={"ID":"9cd3314f-e47e-4589-8af1-9773667589de","Type":"ContainerDied","Data":"205c2d2bbd944d18e81b5b9f44ee50ebe83659bfbf074f067679ee85eeddcf98"}
Sep 08 16:37:59.708508 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.708478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:59.828499 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.828442    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:37:59.846597 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.846565    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:37:59.918742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.918706    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca\") pod \"9cd3314f-e47e-4589-8af1-9773667589de\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") "
Sep 08 16:37:59.918952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.918754    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls\") pod \"9cd3314f-e47e-4589-8af1-9773667589de\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") "
Sep 08 16:37:59.918952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.918832    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfpcj\" (UniqueName: \"kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj\") pod \"9cd3314f-e47e-4589-8af1-9773667589de\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") "
Sep 08 16:37:59.918952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.918894    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs\") pod \"9cd3314f-e47e-4589-8af1-9773667589de\" (UID: \"9cd3314f-e47e-4589-8af1-9773667589de\") "
Sep 08 16:37:59.919127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.919048    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cd3314f-e47e-4589-8af1-9773667589de" (UID: "9cd3314f-e47e-4589-8af1-9773667589de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:37:59.919612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.919585    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd3314f-e47e-4589-8af1-9773667589de-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:37:59.921373 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.921343    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9cd3314f-e47e-4589-8af1-9773667589de" (UID: "9cd3314f-e47e-4589-8af1-9773667589de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:37:59.921672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.921646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cd3314f-e47e-4589-8af1-9773667589de" (UID: "9cd3314f-e47e-4589-8af1-9773667589de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:37:59.921757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.921666    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj" (OuterVolumeSpecName: "kube-api-access-pfpcj") pod "9cd3314f-e47e-4589-8af1-9773667589de" (UID: "9cd3314f-e47e-4589-8af1-9773667589de"). InnerVolumeSpecName "kube-api-access-pfpcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:37:59.953908 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:37:59.953883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:00.027144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.027111    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.027144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.027140    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfpcj\" (UniqueName: \"kubernetes.io/projected/9cd3314f-e47e-4589-8af1-9773667589de-kube-api-access-pfpcj\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.027349 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.027151    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd3314f-e47e-4589-8af1-9773667589de-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.409924 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.409873    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" podUID="709ade1a-23b5-427c-a71d-85ae9ce22119" containerName="main" containerID="cri-o://01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d" gracePeriod=45
Sep 08 16:38:00.662970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.662917    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:38:00.703117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.703094    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"
Sep 08 16:38:00.703241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.703134    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q" event={"ID":"9cd3314f-e47e-4589-8af1-9773667589de","Type":"ContainerDied","Data":"6cba53455dc232cc4401215b4c78a5f98cb59b4665bb3cd6b37ae472de90bd98"}
Sep 08 16:38:00.703324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.703246    2571 scope.go:117] "RemoveContainer" containerID="205c2d2bbd944d18e81b5b9f44ee50ebe83659bfbf074f067679ee85eeddcf98"
Sep 08 16:38:00.704560 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.704534    2571 generic.go:356] "Generic (PLEG): container finished" podID="709ade1a-23b5-427c-a71d-85ae9ce22119" containerID="01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d" exitCode=0
Sep 08 16:38:00.704646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.704584    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"
Sep 08 16:38:00.704702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.704577    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" event={"ID":"709ade1a-23b5-427c-a71d-85ae9ce22119","Type":"ContainerDied","Data":"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"}
Sep 08 16:38:00.704749 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.704701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm" event={"ID":"709ade1a-23b5-427c-a71d-85ae9ce22119","Type":"ContainerDied","Data":"56f6030d8f18751c5c94a60759494e1b69affc89c561eaab145744b5f1f96906"}
Sep 08 16:38:00.714048 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.714030    2571 scope.go:117] "RemoveContainer" containerID="01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"
Sep 08 16:38:00.721860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.721846    2571 scope.go:117] "RemoveContainer" containerID="01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"
Sep 08 16:38:00.722124 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:38:00.722106    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d\": container with ID starting with 01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d not found: ID does not exist" containerID="01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"
Sep 08 16:38:00.722168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.722133    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d"} err="failed to get container status \"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d\": rpc error: code = NotFound desc = could not find container \"01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d\": container with ID starting with 01c8e86795d00ecbb38683f176a86bd67261a4858e0c8fe3abe9bfcbaeceb13d not found: ID does not exist"
Sep 08 16:38:00.729657 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.729635    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:38:00.732757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.732738    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1fa1601c-79ff6dc869-xdh5q"]
Sep 08 16:38:00.733010 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.732992    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733022    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733051    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733096    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcb2n\" (UniqueName: \"kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733114    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733170    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm\") pod \"709ade1a-23b5-427c-a71d-85ae9ce22119\" (UID: \"709ade1a-23b5-427c-a71d-85ae9ce22119\") "
Sep 08 16:38:00.733315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733245    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache" (OuterVolumeSpecName: "model-cache") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:00.733315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733259    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home" (OuterVolumeSpecName: "home") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:00.733590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733336    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:00.733590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733514    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.733590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733526    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.733590 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.733536    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.735448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.735411    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n" (OuterVolumeSpecName: "kube-api-access-xcb2n") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "kube-api-access-xcb2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:00.735448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.735421    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm" (OuterVolumeSpecName: "dshm") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:00.735448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.735421    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "709ade1a-23b5-427c-a71d-85ae9ce22119" (UID: "709ade1a-23b5-427c-a71d-85ae9ce22119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:38:00.834337 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.834310    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcb2n\" (UniqueName: \"kubernetes.io/projected/709ade1a-23b5-427c-a71d-85ae9ce22119-kube-api-access-xcb2n\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.834337 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.834334    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709ade1a-23b5-427c-a71d-85ae9ce22119-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.834475 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.834344    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709ade1a-23b5-427c-a71d-85ae9ce22119-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.887940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:00.887913    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd3314f-e47e-4589-8af1-9773667589de" path="/var/lib/kubelet/pods/9cd3314f-e47e-4589-8af1-9773667589de/volumes"
Sep 08 16:38:01.020436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:01.020356    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:38:01.028312 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:01.028282    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/auth-test-model-e22de8-kserve-78b8fb859f-pfhbm"]
Sep 08 16:38:02.888564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:02.888533    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709ade1a-23b5-427c-a71d-85ae9ce22119" path="/var/lib/kubelet/pods/709ade1a-23b5-427c-a71d-85ae9ce22119/volumes"
Sep 08 16:38:02.988393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:02.988355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:04.970683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:04.970640    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:05.977918 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:05.977875    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:38:07.971057 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:07.971016    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:38:08.066742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:08.066709    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:13.886480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886400    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:38:13.886880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886791    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709ade1a-23b5-427c-a71d-85ae9ce22119" containerName="main"
Sep 08 16:38:13.886880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886804    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="709ade1a-23b5-427c-a71d-85ae9ce22119" containerName="main"
Sep 08 16:38:13.886880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd3314f-e47e-4589-8af1-9773667589de" containerName="maas-api"
Sep 08 16:38:13.886880 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886830    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd3314f-e47e-4589-8af1-9773667589de" containerName="maas-api"
Sep 08 16:38:13.887093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886900    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="709ade1a-23b5-427c-a71d-85ae9ce22119" containerName="main"
Sep 08 16:38:13.887093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.886911    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd3314f-e47e-4589-8af1-9773667589de" containerName="maas-api"
Sep 08 16:38:13.891551 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.891529    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.894124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.894098    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"openshift-service-ca.crt\""
Sep 08 16:38:13.895086 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.895063    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"kube-root-ca.crt\""
Sep 08 16:38:13.895193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.895164    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"sub-test-model-bce20b-kserve-self-signed-certs\""
Sep 08 16:38:13.895193 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.895172    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"default-dockercfg-6sw9q\""
Sep 08 16:38:13.898821 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.898797    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:38:13.931658 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.931620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:13.955760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.955725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.955943 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.955893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.955943 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.955921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.956060 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.955975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.956060 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.956027    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlvj\" (UniqueName: \"kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:13.956163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:13.956122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056529 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056691 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056740 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056805 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlvj\" (UniqueName: \"kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.056970 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.056921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.057125 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.057094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.059022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.058999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.059372 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.059357    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.065562 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.065541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlvj\" (UniqueName: \"kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj\") pod \"sub-test-model-bce20b-kserve-657d58467f-pcngp\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.203117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.203037    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:14.474843 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.474811    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:38:14.477484 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:38:14.477456    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c8526c2_069b_4deb_a8bd_e300aa5a807a.slice/crio-036ddd16dd78e14573ec5c655f001e81aada89f27c3cf45b9993f4193dff2e92 WatchSource:0}: Error finding container 036ddd16dd78e14573ec5c655f001e81aada89f27c3cf45b9993f4193dff2e92: Status 404 returned error can't find the container with id 036ddd16dd78e14573ec5c655f001e81aada89f27c3cf45b9993f4193dff2e92
Sep 08 16:38:14.773496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.773411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" event={"ID":"9c8526c2-069b-4deb-a8bd-e300aa5a807a","Type":"ContainerStarted","Data":"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"}
Sep 08 16:38:14.773496 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.773461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" event={"ID":"9c8526c2-069b-4deb-a8bd-e300aa5a807a","Type":"ContainerStarted","Data":"036ddd16dd78e14573ec5c655f001e81aada89f27c3cf45b9993f4193dff2e92"}
Sep 08 16:38:14.792082 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:14.792036    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" podStartSLOduration=1.7920200309999998 podStartE2EDuration="1.792020031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:14.789393248 +0000 UTC m=+1524.433469940" watchObservedRunningTime="2026-09-08 16:38:14.792020031 +0000 UTC m=+1524.436096723"
Sep 08 16:38:17.897717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:17.897682    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:38:22.991443 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:22.991405    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" podUID="94f22a52-3b9c-4bcc-b31f-720e4180487c" containerName="main" containerID="cri-o://3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2" gracePeriod=45
Sep 08 16:38:23.249392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.249331    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:38:23.345133 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345102    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345178    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345199    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345254    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345282    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5l5z\" (UniqueName: \"kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z\") pod \"94f22a52-3b9c-4bcc-b31f-720e4180487c\" (UID: \"94f22a52-3b9c-4bcc-b31f-720e4180487c\") "
Sep 08 16:38:23.345561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345410    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache" (OuterVolumeSpecName: "model-cache") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:23.345561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345486    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home" (OuterVolumeSpecName: "home") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:23.345561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345498    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:23.345748 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345730    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.345826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345751    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.345826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.345761    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.347406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.347381    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:38:23.347940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.347917    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm" (OuterVolumeSpecName: "dshm") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:23.347940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.347924    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z" (OuterVolumeSpecName: "kube-api-access-z5l5z") pod "94f22a52-3b9c-4bcc-b31f-720e4180487c" (UID: "94f22a52-3b9c-4bcc-b31f-720e4180487c"). InnerVolumeSpecName "kube-api-access-z5l5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:23.446802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.446746    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f22a52-3b9c-4bcc-b31f-720e4180487c-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.446802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.446798    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f22a52-3b9c-4bcc-b31f-720e4180487c-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.446802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.446809    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l5z\" (UniqueName: \"kubernetes.io/projected/94f22a52-3b9c-4bcc-b31f-720e4180487c-kube-api-access-z5l5z\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:38:23.816378 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.816347    2571 generic.go:356] "Generic (PLEG): container finished" podID="94f22a52-3b9c-4bcc-b31f-720e4180487c" containerID="3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2" exitCode=0
Sep 08 16:38:23.816523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.816391    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"
Sep 08 16:38:23.816523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.816404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" event={"ID":"94f22a52-3b9c-4bcc-b31f-720e4180487c","Type":"ContainerDied","Data":"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"}
Sep 08 16:38:23.816523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.816435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj" event={"ID":"94f22a52-3b9c-4bcc-b31f-720e4180487c","Type":"ContainerDied","Data":"2f9d6cbc7fbb44d5b17a24e211559478abdc07fbe796a4e8027dab3a3a4b981e"}
Sep 08 16:38:23.816523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.816453    2571 scope.go:117] "RemoveContainer" containerID="3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"
Sep 08 16:38:23.825613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.825586    2571 scope.go:117] "RemoveContainer" containerID="3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"
Sep 08 16:38:23.825917 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:38:23.825900    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2\": container with ID starting with 3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2 not found: ID does not exist" containerID="3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"
Sep 08 16:38:23.825987 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.825926    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2"} err="failed to get container status \"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2\": rpc error: code = NotFound desc = could not find container \"3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2\": container with ID starting with 3cc480c670d4cd5e085c9087465a239f70457f561cfc11559d4a08aac3bb7fe2 not found: ID does not exist"
Sep 08 16:38:23.839085 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.839062    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:38:23.841933 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:23.841908    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1fa1601c/e2e-lifecycle-model-1fa1601c-kserve-8d7f8c495-rxrxj"]
Sep 08 16:38:24.204148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:24.204058    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:24.204148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:24.204107    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:24.216186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:24.216161    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:24.832809 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:24.832763    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:38:24.889320 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:24.889294    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f22a52-3b9c-4bcc-b31f-720e4180487c" path="/var/lib/kubelet/pods/94f22a52-3b9c-4bcc-b31f-720e4180487c/volumes"
Sep 08 16:38:33.024637 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:33.024597    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:34.178889 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.178852    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:38:34.179243 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.179215    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f22a52-3b9c-4bcc-b31f-720e4180487c" containerName="main"
Sep 08 16:38:34.179243 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.179227    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f22a52-3b9c-4bcc-b31f-720e4180487c" containerName="main"
Sep 08 16:38:34.179314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.179298    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f22a52-3b9c-4bcc-b31f-720e4180487c" containerName="main"
Sep 08 16:38:34.183705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.183683    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.186250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.186230    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"kube-root-ca.crt\""
Sep 08 16:38:34.187320 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.187296    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"openshift-service-ca.crt\""
Sep 08 16:38:34.187410 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.187342    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"sub-test-model-e22de8-kserve-self-signed-certs\""
Sep 08 16:38:34.187574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.187557    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"default-dockercfg-kphbb\""
Sep 08 16:38:34.193170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.193153    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:38:34.210274 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.210248    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:34.239986 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.239965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.240087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.240011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.240087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.240044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.240211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.240118    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.240211 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.240171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.240303 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.240252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxxh\" (UniqueName: \"kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341066 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxxh\" (UniqueName: \"kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341293 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341396 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.341857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.341818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.343631 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.343606    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.344404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.344366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.350795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.350754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxxh\" (UniqueName: \"kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh\") pod \"sub-test-model-e22de8-kserve-6459ccc999-59ktl\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.494786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.494701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:34.766572 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.766493    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:38:34.769761 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:38:34.769735    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbb5454_628d_428b_8585_8472d8e2153a.slice/crio-a771cbcfc574876e9b1947f6443b78468c735d94345b21db5f26e69c1161d2db WatchSource:0}: Error finding container a771cbcfc574876e9b1947f6443b78468c735d94345b21db5f26e69c1161d2db: Status 404 returned error can't find the container with id a771cbcfc574876e9b1947f6443b78468c735d94345b21db5f26e69c1161d2db
Sep 08 16:38:34.872233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.872202    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" event={"ID":"5fbb5454-628d-428b-8585-8472d8e2153a","Type":"ContainerStarted","Data":"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"}
Sep 08 16:38:34.872233 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.872243    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" event={"ID":"5fbb5454-628d-428b-8585-8472d8e2153a","Type":"ContainerStarted","Data":"a771cbcfc574876e9b1947f6443b78468c735d94345b21db5f26e69c1161d2db"}
Sep 08 16:38:34.892637 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:34.892579    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" podStartSLOduration=0.892558514 podStartE2EDuration="892.558514ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:34.889392223 +0000 UTC m=+1544.533468916" watchObservedRunningTime="2026-09-08 16:38:34.892558514 +0000 UTC m=+1544.536635208"
Sep 08 16:38:39.074112 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:39.074078    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:38:44.495003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:44.494964    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:44.495437 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:44.495014    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:44.509465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:44.509421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:44.876261 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:44.876230    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:44.932098 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:44.932072    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:38:45.319106 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:45.319076    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:50.389731 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:50.389693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:38:50.931945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:38:50.931905    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:05.652025 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:05.651988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:05.754975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:05.754943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:05.917040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:05.916945    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:07.171905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:07.171864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:07.272719 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:07.272688    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:07.372865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:07.372830    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:07.979612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:07.979574    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:08.079117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:08.079090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:08.178419 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:08.178349    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:09.437823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:09.437784    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:11.385414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:11.385379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:39:15.178012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:15.177968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:15.252221 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:15.252181    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:15.852056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:15.852023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:22.454586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:22.454545    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:22.566444 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:22.566411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:22.671255 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:22.671215    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:24.361326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:24.361296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:26.379984 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:26.379951    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:39:33.172950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:33.172909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:33.368225 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:33.368188    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:37.608203 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:37.608165    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:39:37.627061 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:37.627023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:38.655565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:38.655527    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:39:38.685206 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:38.685177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:40.810270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:40.810235    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:39:52.629289 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.629229    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" podUID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" containerName="main" containerID="cri-o://e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721" gracePeriod=45
Sep 08 16:39:52.863938 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.863915    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:39:52.961736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961662    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961700    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhlvj\" (UniqueName: \"kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961740    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961827    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961871    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961946    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home\") pod \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\" (UID: \"9c8526c2-069b-4deb-a8bd-e300aa5a807a\") "
Sep 08 16:39:52.961967 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.961946    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:52.962217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.962168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:52.962270 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.962208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home" (OuterVolumeSpecName: "home") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:52.962632 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.962608    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:52.962741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.962635    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:52.962741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.962649    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:52.964131 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.964110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm" (OuterVolumeSpecName: "dshm") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:52.964530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.964504    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:39:52.964618 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:52.964506    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj" (OuterVolumeSpecName: "kube-api-access-bhlvj") pod "9c8526c2-069b-4deb-a8bd-e300aa5a807a" (UID: "9c8526c2-069b-4deb-a8bd-e300aa5a807a"). InnerVolumeSpecName "kube-api-access-bhlvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:39:53.063840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.063810    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8526c2-069b-4deb-a8bd-e300aa5a807a-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:53.063840 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.063835    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlvj\" (UniqueName: \"kubernetes.io/projected/9c8526c2-069b-4deb-a8bd-e300aa5a807a-kube-api-access-bhlvj\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:53.063997 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.063846    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8526c2-069b-4deb-a8bd-e300aa5a807a-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:53.246595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.246524    2571 generic.go:356] "Generic (PLEG): container finished" podID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" containerID="e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721" exitCode=0
Sep 08 16:39:53.246595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.246555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" event={"ID":"9c8526c2-069b-4deb-a8bd-e300aa5a807a","Type":"ContainerDied","Data":"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"}
Sep 08 16:39:53.246595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.246579    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"
Sep 08 16:39:53.246854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.246599    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp" event={"ID":"9c8526c2-069b-4deb-a8bd-e300aa5a807a","Type":"ContainerDied","Data":"036ddd16dd78e14573ec5c655f001e81aada89f27c3cf45b9993f4193dff2e92"}
Sep 08 16:39:53.246854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.246627    2571 scope.go:117] "RemoveContainer" containerID="e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"
Sep 08 16:39:53.255905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.255890    2571 scope.go:117] "RemoveContainer" containerID="e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"
Sep 08 16:39:53.256161 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:39:53.256143    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721\": container with ID starting with e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721 not found: ID does not exist" containerID="e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"
Sep 08 16:39:53.256220 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.256173    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721"} err="failed to get container status \"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721\": rpc error: code = NotFound desc = could not find container \"e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721\": container with ID starting with e14822851d95499236a9c720fac32a7672cee20f5428faaf53d62f6b4ebba721 not found: ID does not exist"
Sep 08 16:39:53.268902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.268792    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:39:53.270748 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.270728    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/sub-test-model-bce20b-kserve-657d58467f-pcngp"]
Sep 08 16:39:53.675378 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.675328    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" podUID="5fbb5454-628d-428b-8585-8472d8e2153a" containerName="main" containerID="cri-o://adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c" gracePeriod=45
Sep 08 16:39:53.925085 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.925062    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:39:53.972198 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972134    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cxxh\" (UniqueName: \"kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972273    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972296    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972369    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:53.972480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972391    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir\") pod \"5fbb5454-628d-428b-8585-8472d8e2153a\" (UID: \"5fbb5454-628d-428b-8585-8472d8e2153a\") "
Sep 08 16:39:53.972635 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972605    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home" (OuterVolumeSpecName: "home") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:53.972708 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972685    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:53.972762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972705    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:53.972762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.972723    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:53.974414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.974388    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:39:53.974414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.974404    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm" (OuterVolumeSpecName: "dshm") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:53.974810 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:53.974788    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh" (OuterVolumeSpecName: "kube-api-access-6cxxh") pod "5fbb5454-628d-428b-8585-8472d8e2153a" (UID: "5fbb5454-628d-428b-8585-8472d8e2153a"). InnerVolumeSpecName "kube-api-access-6cxxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:39:54.073244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.073219    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:54.073244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.073243    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb5454-628d-428b-8585-8472d8e2153a-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:54.073395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.073254    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb5454-628d-428b-8585-8472d8e2153a-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:54.073395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.073262    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxxh\" (UniqueName: \"kubernetes.io/projected/5fbb5454-628d-428b-8585-8472d8e2153a-kube-api-access-6cxxh\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:39:54.253513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.253438    2571 generic.go:356] "Generic (PLEG): container finished" podID="5fbb5454-628d-428b-8585-8472d8e2153a" containerID="adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c" exitCode=0
Sep 08 16:39:54.253513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.253482    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"
Sep 08 16:39:54.253513 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.253489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" event={"ID":"5fbb5454-628d-428b-8585-8472d8e2153a","Type":"ContainerDied","Data":"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"}
Sep 08 16:39:54.253785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.253519    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl" event={"ID":"5fbb5454-628d-428b-8585-8472d8e2153a","Type":"ContainerDied","Data":"a771cbcfc574876e9b1947f6443b78468c735d94345b21db5f26e69c1161d2db"}
Sep 08 16:39:54.253785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.253535    2571 scope.go:117] "RemoveContainer" containerID="adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"
Sep 08 16:39:54.262380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.262365    2571 scope.go:117] "RemoveContainer" containerID="adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"
Sep 08 16:39:54.262614 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:39:54.262595    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c\": container with ID starting with adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c not found: ID does not exist" containerID="adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"
Sep 08 16:39:54.262687 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.262624    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c"} err="failed to get container status \"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c\": rpc error: code = NotFound desc = could not find container \"adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c\": container with ID starting with adae76a0329c0f4dcc8ae659997bfcd9911d558394a4cdbce1b8de52f7364d4c not found: ID does not exist"
Sep 08 16:39:54.278750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.278730    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:39:54.285323 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.285302    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/sub-test-model-e22de8-kserve-6459ccc999-59ktl"]
Sep 08 16:39:54.324244 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.324223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:54.761568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.761534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:39:54.887484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.887453    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbb5454-628d-428b-8585-8472d8e2153a" path="/var/lib/kubelet/pods/5fbb5454-628d-428b-8585-8472d8e2153a/volumes"
Sep 08 16:39:54.887842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:39:54.887829    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" path="/var/lib/kubelet/pods/9c8526c2-069b-4deb-a8bd-e300aa5a807a/volumes"
Sep 08 16:40:00.654626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:00.654589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:00.756497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:00.756464    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:01.001784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:01.001694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:01.341656 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:01.341624    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:01.729926 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:01.729834    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:04.567040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:04.567002    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:04.669538 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:04.669503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:05.697539 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.697503    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:40:05.698062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698043    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" containerName="main"
Sep 08 16:40:05.698143 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698065    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" containerName="main"
Sep 08 16:40:05.698143 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698091    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbb5454-628d-428b-8585-8472d8e2153a" containerName="main"
Sep 08 16:40:05.698143 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698099    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbb5454-628d-428b-8585-8472d8e2153a" containerName="main"
Sep 08 16:40:05.698288 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698200    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbb5454-628d-428b-8585-8472d8e2153a" containerName="main"
Sep 08 16:40:05.698288 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.698214    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8526c2-069b-4deb-a8bd-e300aa5a807a" containerName="main"
Sep 08 16:40:05.702617 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.702595    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.706155 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.706133    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-402a97-serving-cert\""
Sep 08 16:40:05.712867 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.712846    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:40:05.735497 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.735464    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:05.770720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.770693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbw2\" (UniqueName: \"kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.770855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.770745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.770855 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.770848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.770949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.770891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.836217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.836186    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:05.871304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.871276    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbw2\" (UniqueName: \"kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.871421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.871328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.871421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.871374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.871421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.871408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.871551 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:40:05.871472    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-serving-cert: secret "maas-api-e2e-api-a-402a97-serving-cert" not found
Sep 08 16:40:05.871551 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:40:05.871537    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls podName:94a20f4a-0ccd-471d-966d-d9bef18029ec nodeName:}" failed. No retries permitted until 2026-09-08 16:40:06.371523371 +0000 UTC m=+1636.015600041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls") pod "maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" (UID: "94a20f4a-0ccd-471d-966d-d9bef18029ec") : secret "maas-api-e2e-api-a-402a97-serving-cert" not found
Sep 08 16:40:05.872052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.872002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.873837 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.873817    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:05.881869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:05.881849    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbw2\" (UniqueName: \"kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:06.035744 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.035664    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:06.242126 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.242093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:06.348156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.348113    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:06.377321 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.377293    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:06.380306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.380282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") pod \"maas-api-e2e-api-a-402a97-649fcb9456-6wsq6\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:06.614406 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.614317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:06.895050 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.894859    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:40:06.897963 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:40:06.897939    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a20f4a_0ccd_471d_966d_d9bef18029ec.slice/crio-04d99e834fef8725783c63e9ee814302ca4435b7b0b956e69d93fb989117713d WatchSource:0}: Error finding container 04d99e834fef8725783c63e9ee814302ca4435b7b0b956e69d93fb989117713d: Status 404 returned error can't find the container with id 04d99e834fef8725783c63e9ee814302ca4435b7b0b956e69d93fb989117713d
Sep 08 16:40:06.899577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:06.899561    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:40:07.317238 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:07.317203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" event={"ID":"94a20f4a-0ccd-471d-966d-d9bef18029ec","Type":"ContainerStarted","Data":"04d99e834fef8725783c63e9ee814302ca4435b7b0b956e69d93fb989117713d"}
Sep 08 16:40:08.323835 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:08.323795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" event={"ID":"94a20f4a-0ccd-471d-966d-d9bef18029ec","Type":"ContainerStarted","Data":"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"}
Sep 08 16:40:08.324237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:08.323868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:08.343759 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:08.343710    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" podStartSLOduration=2.805414246 podStartE2EDuration="3.34369435s" totalImagesPullingTime="538.280104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:08.341793654 +0000 UTC m=+1637.985870342" watchObservedRunningTime="2026-09-08 16:40:08.34369435 +0000 UTC m=+1637.987771069"
Sep 08 16:40:12.779947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:12.779911    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:40:14.334451 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:14.334424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:40:21.628738 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:21.628701    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:26.661992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.661952    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:26.881505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.881475    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:40:26.884986 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.884964    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:26.887904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.887884    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"default-dockercfg-6sw9q\""
Sep 08 16:40:26.887998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.887960    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"kube-root-ca.crt\""
Sep 08 16:40:26.888832 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.888816    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"rate-test-model-bce20b-kserve-self-signed-certs\""
Sep 08 16:40:26.888832 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.888822    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"openshift-service-ca.crt\""
Sep 08 16:40:26.894465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.894444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:40:26.907250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:26.907225    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:27.065254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.065254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065221    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.065254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.065254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.065477 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rzp\" (UniqueName: \"kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.065477 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.065370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166609 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rzp\" (UniqueName: \"kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.166806 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.166799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.167096 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.167051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.167154 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.167119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.167208 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.167160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.169023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.168999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.169148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.169131    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.174392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.174374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rzp\" (UniqueName: \"kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp\") pod \"rate-test-model-bce20b-kserve-56f5688c95-jqnqs\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.195505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.195479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:27.472981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:27.472942    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:40:27.475570 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:40:27.475543    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b5f527b_7fdd_4f2d_84ca_1555a52174e8.slice/crio-1700d142dd3ecd321403b3a87f37cbe9bd20820ca319b25b18118e7c87d37439 WatchSource:0}: Error finding container 1700d142dd3ecd321403b3a87f37cbe9bd20820ca319b25b18118e7c87d37439: Status 404 returned error can't find the container with id 1700d142dd3ecd321403b3a87f37cbe9bd20820ca319b25b18118e7c87d37439
Sep 08 16:40:28.422819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:28.422762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" event={"ID":"4b5f527b-7fdd-4f2d-84ca-1555a52174e8","Type":"ContainerStarted","Data":"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"}
Sep 08 16:40:28.423186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:28.422825    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" event={"ID":"4b5f527b-7fdd-4f2d-84ca-1555a52174e8","Type":"ContainerStarted","Data":"1700d142dd3ecd321403b3a87f37cbe9bd20820ca319b25b18118e7c87d37439"}
Sep 08 16:40:28.441222 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:28.441177    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" podStartSLOduration=2.44116167 podStartE2EDuration="2.44116167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:28.439276134 +0000 UTC m=+1658.083352824" watchObservedRunningTime="2026-09-08 16:40:28.44116167 +0000 UTC m=+1658.085238360"
Sep 08 16:40:29.534518 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:29.534483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:29.975292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:29.975260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:30.286140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:30.286045    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:30.660593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:30.660560    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:31.182990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:31.182957    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:40:33.893605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:33.893557    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:33.995009 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:33.994968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.071115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.071083    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.236379 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.236344    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.456819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.456716    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.670489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.670454    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.763038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.762959    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:35.774151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.774122    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:40:35.777709 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.777691    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.780651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.780629    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e7d9b1\""
Sep 08 16:40:35.780902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.780884    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e7d9b1-dockercfg-dfd6j\""
Sep 08 16:40:35.786812 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.786791    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:40:35.801423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.801399    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:40:35.804557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.804540    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.813555 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.813530    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:40:35.851708 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.851683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.851818 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.851734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlk9\" (UniqueName: \"kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.851897 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.851877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fnq\" (UniqueName: \"kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.851935 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.851908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.952540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.952509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlk9\" (UniqueName: \"kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.952737 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.952714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fnq\" (UniqueName: \"kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.952863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.952751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.952929 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.952902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.953422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.953400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.953552 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.953533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:35.963505 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.963474    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlk9\" (UniqueName: \"kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9\") pod \"payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:35.963592 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:35.963573    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fnq\" (UniqueName: \"kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq\") pod \"payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:36.089751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.089728    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:36.114833 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.114811    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:36.367007 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.366973    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:40:36.368862 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:40:36.368837    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863925cf_1a57_49c6_8555_9eace5bb09a3.slice/crio-da7d7a18f723106a6e12fa0ca6de3ad446420b5424be5c3ef9cac4aee07c9606 WatchSource:0}: Error finding container da7d7a18f723106a6e12fa0ca6de3ad446420b5424be5c3ef9cac4aee07c9606: Status 404 returned error can't find the container with id da7d7a18f723106a6e12fa0ca6de3ad446420b5424be5c3ef9cac4aee07c9606
Sep 08 16:40:36.402541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.402520    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:40:36.404727 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:40:36.404698    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbdce93_9bf7_4c66_9b80_905cd8e0a9d0.slice/crio-be5764f877049fd6b796f70e8309f1bf0efd29f47f31685feffc56336b3b915b WatchSource:0}: Error finding container be5764f877049fd6b796f70e8309f1bf0efd29f47f31685feffc56336b3b915b: Status 404 returned error can't find the container with id be5764f877049fd6b796f70e8309f1bf0efd29f47f31685feffc56336b3b915b
Sep 08 16:40:36.462204 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.462178    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" event={"ID":"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0","Type":"ContainerStarted","Data":"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"}
Sep 08 16:40:36.462343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.462251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" event={"ID":"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0","Type":"ContainerStarted","Data":"be5764f877049fd6b796f70e8309f1bf0efd29f47f31685feffc56336b3b915b"}
Sep 08 16:40:36.462343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.462309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:36.464212 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.464186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" event={"ID":"863925cf-1a57-49c6-8555-9eace5bb09a3","Type":"ContainerStarted","Data":"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"}
Sep 08 16:40:36.464306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.464233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" event={"ID":"863925cf-1a57-49c6-8555-9eace5bb09a3","Type":"ContainerStarted","Data":"da7d7a18f723106a6e12fa0ca6de3ad446420b5424be5c3ef9cac4aee07c9606"}
Sep 08 16:40:36.464306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.464262    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:36.483352 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.483313    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" podStartSLOduration=1.4832985650000001 podStartE2EDuration="1.483298565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:36.481235433 +0000 UTC m=+1666.125312124" watchObservedRunningTime="2026-09-08 16:40:36.483298565 +0000 UTC m=+1666.127375259"
Sep 08 16:40:36.499052 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:36.499004    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" podStartSLOduration=1.49899129 podStartE2EDuration="1.49899129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:36.498262529 +0000 UTC m=+1666.142339220" watchObservedRunningTime="2026-09-08 16:40:36.49899129 +0000 UTC m=+1666.143067980"
Sep 08 16:40:37.195841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:37.195807    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:37.196279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:37.196072    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:37.208436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:37.208407    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:37.483044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:37.482965    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:40:42.384615 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:42.384572    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:40:47.471097 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:47.471063    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:40:47.471536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:47.471145    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:40:49.716909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:49.716875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:56.076436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:56.076394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:40:58.761879 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:40:58.761841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:01.887007 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:01.886969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:06.398014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:06.397971    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:09.585298 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:09.585266    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:41:17.200912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.200829    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:41:17.201355 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.201017    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" podUID="94a20f4a-0ccd-471d-966d-d9bef18029ec" containerName="maas-api" containerID="cri-o://9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad" gracePeriod=30
Sep 08 16:41:17.304077 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.304037    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:17.452309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.452253    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:41:17.464534 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.464506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:17.533035 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.533008    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rbw2\" (UniqueName: \"kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2\") pod \"94a20f4a-0ccd-471d-966d-d9bef18029ec\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") "
Sep 08 16:41:17.533181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.533102    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca\") pod \"94a20f4a-0ccd-471d-966d-d9bef18029ec\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") "
Sep 08 16:41:17.533181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.533144    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") pod \"94a20f4a-0ccd-471d-966d-d9bef18029ec\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") "
Sep 08 16:41:17.533181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.533173    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs\") pod \"94a20f4a-0ccd-471d-966d-d9bef18029ec\" (UID: \"94a20f4a-0ccd-471d-966d-d9bef18029ec\") "
Sep 08 16:41:17.533519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.533486    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94a20f4a-0ccd-471d-966d-d9bef18029ec" (UID: "94a20f4a-0ccd-471d-966d-d9bef18029ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:17.535551 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.535528    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94a20f4a-0ccd-471d-966d-d9bef18029ec" (UID: "94a20f4a-0ccd-471d-966d-d9bef18029ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:41:17.535750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.535718    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94a20f4a-0ccd-471d-966d-d9bef18029ec" (UID: "94a20f4a-0ccd-471d-966d-d9bef18029ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:41:17.535750 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.535733    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2" (OuterVolumeSpecName: "kube-api-access-5rbw2") pod "94a20f4a-0ccd-471d-966d-d9bef18029ec" (UID: "94a20f4a-0ccd-471d-966d-d9bef18029ec"). InnerVolumeSpecName "kube-api-access-5rbw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:17.563720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.563695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:17.634465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.634430    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbw2\" (UniqueName: \"kubernetes.io/projected/94a20f4a-0ccd-471d-966d-d9bef18029ec-kube-api-access-5rbw2\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.634465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.634458    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a20f4a-0ccd-471d-966d-d9bef18029ec-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.634465 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.634470    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.634712 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.634478    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94a20f4a-0ccd-471d-966d-d9bef18029ec-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.637904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.637875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:17.678342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.678315    2571 generic.go:356] "Generic (PLEG): container finished" podID="94a20f4a-0ccd-471d-966d-d9bef18029ec" containerID="9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad" exitCode=0
Sep 08 16:41:17.678458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.678355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" event={"ID":"94a20f4a-0ccd-471d-966d-d9bef18029ec","Type":"ContainerDied","Data":"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"}
Sep 08 16:41:17.678458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.678362    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"
Sep 08 16:41:17.678458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.678396    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6" event={"ID":"94a20f4a-0ccd-471d-966d-d9bef18029ec","Type":"ContainerDied","Data":"04d99e834fef8725783c63e9ee814302ca4435b7b0b956e69d93fb989117713d"}
Sep 08 16:41:17.678458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.678418    2571 scope.go:117] "RemoveContainer" containerID="9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"
Sep 08 16:41:17.688817 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.688793    2571 scope.go:117] "RemoveContainer" containerID="9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"
Sep 08 16:41:17.689109 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:41:17.689087    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad\": container with ID starting with 9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad not found: ID does not exist" containerID="9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"
Sep 08 16:41:17.689186 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.689122    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad"} err="failed to get container status \"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad\": rpc error: code = NotFound desc = could not find container \"9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad\": container with ID starting with 9c25b2e4c6283672e3514a64f22064c81f5b330029cba0cd6536d00da772e8ad not found: ID does not exist"
Sep 08 16:41:17.702953 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.702872    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:41:17.718561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.718536    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-402a97-649fcb9456-6wsq6"]
Sep 08 16:41:17.721791 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:17.721742    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:18.000594 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:18.000506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:18.887894 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:18.887863    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a20f4a-0ccd-471d-966d-d9bef18029ec" path="/var/lib/kubelet/pods/94a20f4a-0ccd-471d-966d-d9bef18029ec/volumes"
Sep 08 16:41:23.674653 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:23.674611    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:41:24.157910 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:24.157872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:24.259393 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:24.259353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:24.434683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:24.434589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:24.869875 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:24.869813    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:33.780304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:33.780273    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:41:40.363479 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.363447    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:40.375422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.375394    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:41:40.375661 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.375636    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" podUID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" containerName="payload-processing" containerID="cri-o://217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c" gracePeriod=30
Sep 08 16:41:40.384246 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.384222    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:41:40.384434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.384405    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" podUID="863925cf-1a57-49c6-8555-9eace5bb09a3" containerName="payload-pre-processing" containerID="cri-o://8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e" gracePeriod=30
Sep 08 16:41:40.463871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.463839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:40.564962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.564927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:40.662745 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.662725    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:41:40.664667 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.664643    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:40.665627 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.665607    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:41:40.757916 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.757881    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume\") pod \"863925cf-1a57-49c6-8555-9eace5bb09a3\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") "
Sep 08 16:41:40.758095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.757969    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28fnq\" (UniqueName: \"kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq\") pod \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") "
Sep 08 16:41:40.758095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.758010    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume\") pod \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\" (UID: \"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0\") "
Sep 08 16:41:40.758095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.758041    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvlk9\" (UniqueName: \"kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9\") pod \"863925cf-1a57-49c6-8555-9eace5bb09a3\" (UID: \"863925cf-1a57-49c6-8555-9eace5bb09a3\") "
Sep 08 16:41:40.758263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.758240    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "863925cf-1a57-49c6-8555-9eace5bb09a3" (UID: "863925cf-1a57-49c6-8555-9eace5bb09a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:40.758367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.758352    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863925cf-1a57-49c6-8555-9eace5bb09a3-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:40.758433 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.758422    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" (UID: "0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:40.760340 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.760316    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9" (OuterVolumeSpecName: "kube-api-access-tvlk9") pod "863925cf-1a57-49c6-8555-9eace5bb09a3" (UID: "863925cf-1a57-49c6-8555-9eace5bb09a3"). InnerVolumeSpecName "kube-api-access-tvlk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:40.760421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.760333    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq" (OuterVolumeSpecName: "kube-api-access-28fnq") pod "0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" (UID: "0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0"). InnerVolumeSpecName "kube-api-access-28fnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:40.789611 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.789581    2571 generic.go:356] "Generic (PLEG): container finished" podID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" containerID="217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c" exitCode=0
Sep 08 16:41:40.789717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.789628    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"
Sep 08 16:41:40.789717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.789645    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" event={"ID":"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0","Type":"ContainerDied","Data":"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"}
Sep 08 16:41:40.789717 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.789691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5" event={"ID":"0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0","Type":"ContainerDied","Data":"be5764f877049fd6b796f70e8309f1bf0efd29f47f31685feffc56336b3b915b"}
Sep 08 16:41:40.789920 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.789722    2571 scope.go:117] "RemoveContainer" containerID="217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"
Sep 08 16:41:40.791006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.790982    2571 generic.go:356] "Generic (PLEG): container finished" podID="863925cf-1a57-49c6-8555-9eace5bb09a3" containerID="8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e" exitCode=0
Sep 08 16:41:40.791074 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.791017    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"
Sep 08 16:41:40.791074 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.791032    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" event={"ID":"863925cf-1a57-49c6-8555-9eace5bb09a3","Type":"ContainerDied","Data":"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"}
Sep 08 16:41:40.791156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.791072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk" event={"ID":"863925cf-1a57-49c6-8555-9eace5bb09a3","Type":"ContainerDied","Data":"da7d7a18f723106a6e12fa0ca6de3ad446420b5424be5c3ef9cac4aee07c9606"}
Sep 08 16:41:40.798394 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.798293    2571 scope.go:117] "RemoveContainer" containerID="217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"
Sep 08 16:41:40.798611 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:41:40.798588    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c\": container with ID starting with 217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c not found: ID does not exist" containerID="217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"
Sep 08 16:41:40.798730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.798616    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c"} err="failed to get container status \"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c\": rpc error: code = NotFound desc = could not find container \"217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c\": container with ID starting with 217ac29c34a1d146b782db71d7ac9b42df6c8a1084ab0312caed71e7a875c93c not found: ID does not exist"
Sep 08 16:41:40.798730 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.798634    2571 scope.go:117] "RemoveContainer" containerID="8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"
Sep 08 16:41:40.806159 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.806140    2571 scope.go:117] "RemoveContainer" containerID="8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"
Sep 08 16:41:40.806421 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:41:40.806402    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e\": container with ID starting with 8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e not found: ID does not exist" containerID="8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"
Sep 08 16:41:40.806485 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.806427    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e"} err="failed to get container status \"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e\": rpc error: code = NotFound desc = could not find container \"8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e\": container with ID starting with 8f4721f233641273fdb54e92478eacd977b4cd51077a3b9bba6ef222d5825a0e not found: ID does not exist"
Sep 08 16:41:40.817604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.817584    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:41:40.822149 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.822129    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7d9b1-65dc4c9765-rfnt5"]
Sep 08 16:41:40.833833 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.833813    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:41:40.839615 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.839597    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7d9b1-675b4fd658-2rbmk"]
Sep 08 16:41:40.859112 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.859094    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28fnq\" (UniqueName: \"kubernetes.io/projected/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-kube-api-access-28fnq\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:40.859168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.859114    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:40.859168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.859124    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlk9\" (UniqueName: \"kubernetes.io/projected/863925cf-1a57-49c6-8555-9eace5bb09a3-kube-api-access-tvlk9\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:40.888172 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.888123    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" path="/var/lib/kubelet/pods/0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0/volumes"
Sep 08 16:41:40.888521 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.888508    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863925cf-1a57-49c6-8555-9eace5bb09a3" path="/var/lib/kubelet/pods/863925cf-1a57-49c6-8555-9eace5bb09a3/volumes"
Sep 08 16:41:40.931181 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:40.931155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:45.480290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:45.480258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:41:47.098728 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:47.098684    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:47.233113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:47.233074    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:47.402253 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:47.402131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:47.830480 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:47.830412    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:52.213124 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:52.213094    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:41:52.234601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:52.234557    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:53.185511 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:53.185465    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:53.581914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:53.581882    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:53.874893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:53.874794    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:54.342110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:54.342082    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:57.225962 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:57.225923    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:57.325894 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:57.325860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:57.979300 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:57.979261    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:41:58.219659 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.219623    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:41:58.219871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.219846    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" podUID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" containerName="maas-api" containerID="cri-o://94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2" gracePeriod=30
Sep 08 16:41:58.253582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.253540    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.367639 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.367600    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.456720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.456693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.472012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.471984    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:41:58.509868 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.509794    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs\") pod \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") "
Sep 08 16:41:58.510016 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.509900    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4ps8\" (UniqueName: \"kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8\") pod \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") "
Sep 08 16:41:58.510016 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.509984    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") pod \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") "
Sep 08 16:41:58.510140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.510047    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca\") pod \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\" (UID: \"ab4af05d-f22a-4453-aa11-c2bbb07026c1\") "
Sep 08 16:41:58.510570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.510526    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab4af05d-f22a-4453-aa11-c2bbb07026c1" (UID: "ab4af05d-f22a-4453-aa11-c2bbb07026c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:58.512760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.512729    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab4af05d-f22a-4453-aa11-c2bbb07026c1" (UID: "ab4af05d-f22a-4453-aa11-c2bbb07026c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:41:58.512891 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.512762    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8" (OuterVolumeSpecName: "kube-api-access-p4ps8") pod "ab4af05d-f22a-4453-aa11-c2bbb07026c1" (UID: "ab4af05d-f22a-4453-aa11-c2bbb07026c1"). InnerVolumeSpecName "kube-api-access-p4ps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:58.513079 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.513056    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab4af05d-f22a-4453-aa11-c2bbb07026c1" (UID: "ab4af05d-f22a-4453-aa11-c2bbb07026c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:41:58.551998 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.551968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.611729 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.611695    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ps8\" (UniqueName: \"kubernetes.io/projected/ab4af05d-f22a-4453-aa11-c2bbb07026c1-kube-api-access-p4ps8\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.611863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.611735    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.611863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.611750    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4af05d-f22a-4453-aa11-c2bbb07026c1-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.611863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.611763    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab4af05d-f22a-4453-aa11-c2bbb07026c1-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.619992 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.619752    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.845888 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.845845    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:58.877633 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.877606    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" containerID="94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2" exitCode=0
Sep 08 16:41:58.877815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.877659    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"
Sep 08 16:41:58.877815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.877699    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" event={"ID":"ab4af05d-f22a-4453-aa11-c2bbb07026c1","Type":"ContainerDied","Data":"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"}
Sep 08 16:41:58.877815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.877749    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd" event={"ID":"ab4af05d-f22a-4453-aa11-c2bbb07026c1","Type":"ContainerDied","Data":"153050ab70e1ab9fba851cc36a03ffee286ce095c625d25a686576cba3299d71"}
Sep 08 16:41:58.877815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.877791    2571 scope.go:117] "RemoveContainer" containerID="94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"
Sep 08 16:41:58.887585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.887570    2571 scope.go:117] "RemoveContainer" containerID="94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"
Sep 08 16:41:58.887855 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:41:58.887836    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2\": container with ID starting with 94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2 not found: ID does not exist" containerID="94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"
Sep 08 16:41:58.887913 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.887863    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2"} err="failed to get container status \"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2\": rpc error: code = NotFound desc = could not find container \"94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2\": container with ID starting with 94c7be292ea8e937c48382c60d627cf14c660472d8bce7710e075ff2c26222e2 not found: ID does not exist"
Sep 08 16:41:58.898048 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.898024    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:41:58.902304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:58.902287    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8ac343-75f74d6766-zvcdd"]
Sep 08 16:41:59.257628 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.257546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:59.400944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.400911    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:41:59.485432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485398    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:41:59.485762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485747    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" containerName="maas-api"
Sep 08 16:41:59.485762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485760    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" containerName="maas-api"
Sep 08 16:41:59.485762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485788    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" containerName="payload-processing"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485797    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" containerName="payload-processing"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485822    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863925cf-1a57-49c6-8555-9eace5bb09a3" containerName="payload-pre-processing"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485828    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="863925cf-1a57-49c6-8555-9eace5bb09a3" containerName="payload-pre-processing"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485840    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a20f4a-0ccd-471d-966d-d9bef18029ec" containerName="maas-api"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485845    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a20f4a-0ccd-471d-966d-d9bef18029ec" containerName="maas-api"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485907    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" containerName="maas-api"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485915    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="863925cf-1a57-49c6-8555-9eace5bb09a3" containerName="payload-pre-processing"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485923    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a20f4a-0ccd-471d-966d-d9bef18029ec" containerName="maas-api"
Sep 08 16:41:59.485980 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.485929    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbdce93-9bf7-4c66-9b80-905cd8e0a9d0" containerName="payload-processing"
Sep 08 16:41:59.490245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.490229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.492882 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.492862    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-352142-dockercfg-8zqv6\""
Sep 08 16:41:59.493001 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.492862    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-352142\""
Sep 08 16:41:59.499989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.499969    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:41:59.521116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.521017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.526868 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.521185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b478q\" (UniqueName: \"kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.622402 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.622366    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b478q\" (UniqueName: \"kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.622573 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.622459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.623108 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.623090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.631314 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.631292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b478q\" (UniqueName: \"kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q\") pod \"payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:41:59.800364 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:41:59.800283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:42:00.070229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.070202    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:42:00.072357 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:42:00.072328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3375a70_d509_4256_9f3c_050a498a17fd.slice/crio-988a0e20f4d1c6dc650a31ffc16b231f81c86cb5108da606ede3af8ebf2b3275 WatchSource:0}: Error finding container 988a0e20f4d1c6dc650a31ffc16b231f81c86cb5108da606ede3af8ebf2b3275: Status 404 returned error can't find the container with id 988a0e20f4d1c6dc650a31ffc16b231f81c86cb5108da606ede3af8ebf2b3275
Sep 08 16:42:00.888736 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.888701    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4af05d-f22a-4453-aa11-c2bbb07026c1" path="/var/lib/kubelet/pods/ab4af05d-f22a-4453-aa11-c2bbb07026c1/volumes"
Sep 08 16:42:00.890258 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.890233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" event={"ID":"d3375a70-d509-4256-9f3c-050a498a17fd","Type":"ContainerStarted","Data":"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"}
Sep 08 16:42:00.890390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.890275    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" event={"ID":"d3375a70-d509-4256-9f3c-050a498a17fd","Type":"ContainerStarted","Data":"988a0e20f4d1c6dc650a31ffc16b231f81c86cb5108da606ede3af8ebf2b3275"}
Sep 08 16:42:00.890390 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.890294    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:42:00.907922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:00.907863    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" podStartSLOduration=1.90784669 podStartE2EDuration="1.90784669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:00.907495162 +0000 UTC m=+1750.551571852" watchObservedRunningTime="2026-09-08 16:42:00.90784669 +0000 UTC m=+1750.551923385"
Sep 08 16:42:03.753568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:03.753534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:04.998512 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:04.998472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:05.097660 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:05.097620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:05.266064 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:05.265989    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:42:05.266217 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:05.266193    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" containerID="cri-o://13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff" gracePeriod=30
Sep 08 16:42:05.294603 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:05.294570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:05.725398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:05.725363    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:07.232637 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.232575    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerName="main" containerID="cri-o://4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c" gracePeriod=45
Sep 08 16:42:07.485151 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.485092    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:42:07.587070 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.587029    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 16:42:07.587229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.587090    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:42:07.590200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590180    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8rzp\" (UniqueName: \"kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590262 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590241    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590308 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590261    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590308 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590299    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590379 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590324    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590411 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590396    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs\") pod \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\" (UID: \"4b5f527b-7fdd-4f2d-84ca-1555a52174e8\") "
Sep 08 16:42:07.590595 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590568    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:07.590751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home" (OuterVolumeSpecName: "home") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:07.590751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:07.590751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590721    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.590751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.590741    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.592644 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.592615    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:42:07.592941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.592917    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm" (OuterVolumeSpecName: "dshm") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:07.593045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.592939    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp" (OuterVolumeSpecName: "kube-api-access-d8rzp") pod "4b5f527b-7fdd-4f2d-84ca-1555a52174e8" (UID: "4b5f527b-7fdd-4f2d-84ca-1555a52174e8"). InnerVolumeSpecName "kube-api-access-d8rzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:07.692237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.692201    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.692237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.692230    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rzp\" (UniqueName: \"kubernetes.io/projected/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-kube-api-access-d8rzp\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.692237 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.692240    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.692447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.692247    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b5f527b-7fdd-4f2d-84ca-1555a52174e8-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:07.923061 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.923027    2571 generic.go:356] "Generic (PLEG): container finished" podID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerID="4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c" exitCode=0
Sep 08 16:42:07.923229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.923073    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"
Sep 08 16:42:07.923229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.923114    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" event={"ID":"4b5f527b-7fdd-4f2d-84ca-1555a52174e8","Type":"ContainerDied","Data":"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"}
Sep 08 16:42:07.923229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.923158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" event={"ID":"4b5f527b-7fdd-4f2d-84ca-1555a52174e8","Type":"ContainerDied","Data":"1700d142dd3ecd321403b3a87f37cbe9bd20820ca319b25b18118e7c87d37439"}
Sep 08 16:42:07.923229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.923179    2571 scope.go:117] "RemoveContainer" containerID="4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"
Sep 08 16:42:07.932201 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.932182    2571 scope.go:117] "RemoveContainer" containerID="4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"
Sep 08 16:42:07.932454 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:42:07.932437    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c\": container with ID starting with 4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c not found: ID does not exist" containerID="4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"
Sep 08 16:42:07.932528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.932460    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c"} err="failed to get container status \"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c\": rpc error: code = NotFound desc = could not find container \"4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c\": container with ID starting with 4ebb458ae0bcc922114f21b8480bf8079d9f678d5391af1e3f0fb0af4279050c not found: ID does not exist"
Sep 08 16:42:07.944230 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.944207    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:42:07.947940 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:07.947919    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs"]
Sep 08 16:42:08.888432 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:08.888389    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" path="/var/lib/kubelet/pods/4b5f527b-7fdd-4f2d-84ca-1555a52174e8/volumes"
Sep 08 16:42:09.782658 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:09.782621    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:42:10.519295 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.519269    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:42:10.617434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617403    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617455    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617507    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617562    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617596    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwg4w\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617661    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617791    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.617914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.617872    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket\") pod \"60b0d3aa-33a2-4061-938c-1ad06acb8801\" (UID: \"60b0d3aa-33a2-4061-938c-1ad06acb8801\") "
Sep 08 16:42:10.618341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618074    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:10.618341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618192    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:42:10.618341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618230    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.618341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618224    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:10.618341 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618333    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:10.618574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.618364    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data" (OuterVolumeSpecName: "istio-data") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:10.619850 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.619828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:42:10.620118 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.620089    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w" (OuterVolumeSpecName: "kube-api-access-fwg4w") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "kube-api-access-fwg4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:10.620246 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.620233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token" (OuterVolumeSpecName: "istio-token") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:10.620310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.620245    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60b0d3aa-33a2-4061-938c-1ad06acb8801" (UID: "60b0d3aa-33a2-4061-938c-1ad06acb8801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:10.718877 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718849    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.718877 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718873    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718882    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718891    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718900    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718907    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718914    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b0d3aa-33a2-4061-938c-1ad06acb8801-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.719040 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.718921    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwg4w\" (UniqueName: \"kubernetes.io/projected/60b0d3aa-33a2-4061-938c-1ad06acb8801-kube-api-access-fwg4w\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.939919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.939887    2571 generic.go:356] "Generic (PLEG): container finished" podID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerID="13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff" exitCode=0
Sep 08 16:42:10.940041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.939931    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"
Sep 08 16:42:10.940041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.939964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerDied","Data":"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"}
Sep 08 16:42:10.940041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.940015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc" event={"ID":"60b0d3aa-33a2-4061-938c-1ad06acb8801","Type":"ContainerDied","Data":"6a2ce1283cc9a776f66161454a6e2c73cc45b67f46402d4c30f2e8ffa33baeea"}
Sep 08 16:42:10.940041 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.940039    2571 scope.go:117] "RemoveContainer" containerID="13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"
Sep 08 16:42:10.948711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.948693    2571 scope.go:117] "RemoveContainer" containerID="22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"
Sep 08 16:42:10.956367 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.956316    2571 scope.go:117] "RemoveContainer" containerID="13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"
Sep 08 16:42:10.956760 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:42:10.956733    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff\": container with ID starting with 13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff not found: ID does not exist" containerID="13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"
Sep 08 16:42:10.956889 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.956798    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff"} err="failed to get container status \"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff\": rpc error: code = NotFound desc = could not find container \"13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff\": container with ID starting with 13c2ae64ce74c8ba5d1ac5a7ef0d602636842d1c20857663890fe50634a470ff not found: ID does not exist"
Sep 08 16:42:10.956889 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.956825    2571 scope.go:117] "RemoveContainer" containerID="22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"
Sep 08 16:42:10.957153 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:42:10.957119    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370\": container with ID starting with 22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370 not found: ID does not exist" containerID="22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"
Sep 08 16:42:10.957218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.957148    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370"} err="failed to get container status \"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370\": rpc error: code = NotFound desc = could not find container \"22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370\": container with ID starting with 22551223f6754395755f37a111a2cf4566db93e7eeb7f2d523c11556ae300370 not found: ID does not exist"
Sep 08 16:42:10.958742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.958721    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:42:10.963062 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:10.963044    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8ac343-openshift-default-c97766f6b-6tmwc"]
Sep 08 16:42:11.896878 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:11.896849    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:42:12.470522 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:12.470483    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bce20b/rate-test-model-bce20b-kserve-56f5688c95-jqnqs" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Sep 08 16:42:12.888188 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:12.888154    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" path="/var/lib/kubelet/pods/60b0d3aa-33a2-4061-938c-1ad06acb8801/volumes"
Sep 08 16:42:16.161744 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:16.161710    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:18.049547 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:18.049519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:19.889058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:19.889021    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:20.682306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:20.682270    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:42:22.153391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:22.153353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:22.779269 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:22.779232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:22.880960 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:22.880925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:23.112045 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.112004    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:23.234546 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234516    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:42:23.234914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234886    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:23.234914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234898    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:23.234914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234910    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerName="main"
Sep 08 16:42:23.234914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234916    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerName="main"
Sep 08 16:42:23.235044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234979    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:23.235044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234987    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5f527b-7fdd-4f2d-84ca-1555a52174e8" containerName="main"
Sep 08 16:42:23.235044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.234992    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:23.237868 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.237845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.260258 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.260235    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-995b22-openshift-default-dockercfg-gjzvw\""
Sep 08 16:42:23.295571 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.295543    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:42:23.305436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.305392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:23.332431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332399    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t875m\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332586 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332581    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332723 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332861 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.332905 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.332854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433366    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433395    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433682 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433682 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433819 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433717    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433832    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.433884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t875m\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.434021 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.434021 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.433984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.434303 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.434284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.434439 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.434419    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.434472 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.434437    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.435811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.435787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.436044 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.436026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.442192 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.442166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.442422 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.442405    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t875m\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m\") pod \"e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") " pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.547527 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.547493    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:23.561171 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.561144    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:23.827889 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.827848    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:42:23.830827 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:42:23.830799    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382116d9_6863_4fb6_b2f7_423891198dd8.slice/crio-5c4e39d792ca92fb6c85ede471c0d80ff3b02e3093cfed90ec7b47d52b53cc19 WatchSource:0}: Error finding container 5c4e39d792ca92fb6c85ede471c0d80ff3b02e3093cfed90ec7b47d52b53cc19: Status 404 returned error can't find the container with id 5c4e39d792ca92fb6c85ede471c0d80ff3b02e3093cfed90ec7b47d52b53cc19
Sep 08 16:42:23.833435 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.833407    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:23.833517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.833470    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:23.833517 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:23.833496    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:24.001827 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:24.001786    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerStarted","Data":"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"}
Sep 08 16:42:24.002009 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:24.001848    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerStarted","Data":"5c4e39d792ca92fb6c85ede471c0d80ff3b02e3093cfed90ec7b47d52b53cc19"}
Sep 08 16:42:24.023779 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:24.023721    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podStartSLOduration=1.023706813 podStartE2EDuration="1.023706813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:24.021709358 +0000 UTC m=+1773.665786049" watchObservedRunningTime="2026-09-08 16:42:24.023706813 +0000 UTC m=+1773.667783504"
Sep 08 16:42:24.548093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:24.548056    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:24.553323 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:24.553297    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:25.006081 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:25.006049    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:25.007095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:25.007072    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:25.996605 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:25.996572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:26.093834 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:26.093796    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:27.138577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:27.138543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:27.244268 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:27.244225    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:27.555218 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:27.555126    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:27.672158 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:27.672124    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:27.768084 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:27.768049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:32.687165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:32.687123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:42:41.047989 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:41.047936    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 16:42:41.048544 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:41.048029    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 16:42:43.095162 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.095132    2571 generic.go:356] "Generic (PLEG): container finished" podID="382116d9-6863-4fb6-b2f7-423891198dd8" containerID="55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c" exitCode=0
Sep 08 16:42:43.095713 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.095211    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerDied","Data":"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"}
Sep 08 16:42:43.095713 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.095633    2571 scope.go:117] "RemoveContainer" containerID="55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"
Sep 08 16:42:43.096693 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.096670    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:43.096742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.096733    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:43.096798 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:43.096781    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:42:44.101749 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:44.101714    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerStarted","Data":"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"}
Sep 08 16:42:44.548313 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:44.548211    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:44.548502 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:44.548471    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 08 16:42:44.548577 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:44.548551    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 08 16:42:45.548213 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:45.548181    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 08 16:42:45.548574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:45.548241    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 08 16:42:46.548705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:46.548661    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 08 16:42:46.549120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:46.548736    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 08 16:42:47.548865 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:47.548827    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 08 16:42:47.549225 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:47.548890    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 08 16:42:48.548111 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:48.548073    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 08 16:42:48.548317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:48.548141    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 08 16:42:49.552148 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:49.552118    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:49.552516 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:49.552370    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:49.552956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:49.552934    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:42:50.872130 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:50.872091    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:50.942696 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:50.942663    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:42:50.946226 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:50.946205    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:42:57.375971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.375937    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:42:57.376326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.376283    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:57.376326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.376294    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b0d3aa-33a2-4061-938c-1ad06acb8801" containerName="istio-proxy"
Sep 08 16:42:57.379308 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.379286    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.383035 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.383008    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-352142\"/\"default-dockercfg-xhf9x\""
Sep 08 16:42:57.383035 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.383008    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-352142\"/\"openshift-service-ca.crt\""
Sep 08 16:42:57.383309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.383290    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-352142\"/\"kube-root-ca.crt\""
Sep 08 16:42:57.383404 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.383355    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-352142\"/\"ipp-route-352142-kserve-self-signed-certs\""
Sep 08 16:42:57.389373 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.389352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:42:57.448977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.448945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.449123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.448994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.449123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.449037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.449123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.449058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.449123 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.449073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.449358 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.449177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njgk\" (UniqueName: \"kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.489309 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.489277    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:42:57.550034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njgk\" (UniqueName: \"kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550043    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550304 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550550 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550616 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.550667 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.550618    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.552490 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.552473    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.552740 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.552724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.558692 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.558663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njgk\" (UniqueName: \"kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk\") pod \"ipp-route-352142-kserve-77f6b56ccf-htqfz\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") " pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.688491 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.688407    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:42:57.960464 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:57.960438    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:42:57.962595 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:42:57.962564    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc185d759_e6b4_471d_9a3f_4fbe0c1aa295.slice/crio-27f55bd411333d65650dcff44e1c5c1397c88a8755565e84b3ca759c53d3dbcc WatchSource:0}: Error finding container 27f55bd411333d65650dcff44e1c5c1397c88a8755565e84b3ca759c53d3dbcc: Status 404 returned error can't find the container with id 27f55bd411333d65650dcff44e1c5c1397c88a8755565e84b3ca759c53d3dbcc
Sep 08 16:42:58.166302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:58.166264    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" event={"ID":"c185d759-e6b4-471d-9a3f-4fbe0c1aa295","Type":"ContainerStarted","Data":"3b8dff7467641ee92e5508c346e11940e088cdb0874dd9d1035ffd5a70e2abe9"}
Sep 08 16:42:58.166489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:58.166309    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" event={"ID":"c185d759-e6b4-471d-9a3f-4fbe0c1aa295","Type":"ContainerStarted","Data":"27f55bd411333d65650dcff44e1c5c1397c88a8755565e84b3ca759c53d3dbcc"}
Sep 08 16:42:58.186284 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:42:58.186242    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" podStartSLOduration=1.186228436 podStartE2EDuration="1.186228436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:58.184856851 +0000 UTC m=+1807.828933545" watchObservedRunningTime="2026-09-08 16:42:58.186228436 +0000 UTC m=+1807.830305126"
Sep 08 16:43:01.074442 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:01.074411    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:43:07.689324 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:07.689281    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:43:07.689823 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:07.689344    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:43:07.701762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:07.701736    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:43:08.223631 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:08.223599    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:43:24.888995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:24.888962    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:43:31.482930 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:31.482895    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9_d3375a70-d509-4256-9f3c-050a498a17fd/payload-processing/0.log"
Sep 08 16:43:32.419296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:32.419260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:32.521037 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:32.521000    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:32.620024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:32.619987    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:35.001364 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:35.001331    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:35.088941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:35.088904    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:35.466440 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:35.466400    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.135965 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.135933    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.196751 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.196717    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.297669 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.297634    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.668902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.668860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.772312 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.772272    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:43:36.803828 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.803792    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:43:36.807359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.807339    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:36.811793 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.811746    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8dca27\""
Sep 08 16:43:36.811899 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.811820    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8dca27-dockercfg-t44dd\""
Sep 08 16:43:36.825234 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.825209    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:43:36.916887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.916851    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:36.917069 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:36.916918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6qt\" (UniqueName: \"kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.017952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.017878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.017952 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.017939    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6qt\" (UniqueName: \"kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.018559 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.018540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.030053 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.030028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6qt\" (UniqueName: \"kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt\") pod \"payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.117857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.117833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:37.387947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:37.387917    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:43:37.391420 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:43:37.391393    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9682c7b_f2d6_47b1_b9ec_39865c26687f.slice/crio-91a468acea0d7bc32eff0e6452bb7cbda279a3a9ff0c504cee729a36b0bd198f WatchSource:0}: Error finding container 91a468acea0d7bc32eff0e6452bb7cbda279a3a9ff0c504cee729a36b0bd198f: Status 404 returned error can't find the container with id 91a468acea0d7bc32eff0e6452bb7cbda279a3a9ff0c504cee729a36b0bd198f
Sep 08 16:43:38.356720 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:38.356678    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" event={"ID":"d9682c7b-f2d6-47b1-b9ec-39865c26687f","Type":"ContainerStarted","Data":"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"}
Sep 08 16:43:38.356915 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:38.356740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" event={"ID":"d9682c7b-f2d6-47b1-b9ec-39865c26687f","Type":"ContainerStarted","Data":"91a468acea0d7bc32eff0e6452bb7cbda279a3a9ff0c504cee729a36b0bd198f"}
Sep 08 16:43:38.356915 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:38.356805    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:38.375458 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:38.375415    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" podStartSLOduration=2.375400209 podStartE2EDuration="2.375400209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:38.373992111 +0000 UTC m=+1848.018068832" watchObservedRunningTime="2026-09-08 16:43:38.375400209 +0000 UTC m=+1848.019476922"
Sep 08 16:43:45.285565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:45.285532    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:43:49.364524 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:49.364493    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:43:59.878945 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:43:59.878895    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:16.919734 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:16.919648    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:16.932331 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:16.932304    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:44:16.932516 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:16.932492    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" podUID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" containerName="payload-processing" containerID="cri-o://6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b" gracePeriod=30
Sep 08 16:44:17.033414 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.033379    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:17.183392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.183360    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:17.190969 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.190949    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:44:17.286115 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.286091    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume\") pod \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") "
Sep 08 16:44:17.286249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.286171    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss6qt\" (UniqueName: \"kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt\") pod \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\" (UID: \"d9682c7b-f2d6-47b1-b9ec-39865c26687f\") "
Sep 08 16:44:17.286554 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.286522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9682c7b-f2d6-47b1-b9ec-39865c26687f" (UID: "d9682c7b-f2d6-47b1-b9ec-39865c26687f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:17.288887 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.288862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt" (OuterVolumeSpecName: "kube-api-access-ss6qt") pod "d9682c7b-f2d6-47b1-b9ec-39865c26687f" (UID: "d9682c7b-f2d6-47b1-b9ec-39865c26687f"). InnerVolumeSpecName "kube-api-access-ss6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:17.340624 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.340596    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:17.387784 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.387732    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9682c7b-f2d6-47b1-b9ec-39865c26687f-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.387933 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.387811    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6qt\" (UniqueName: \"kubernetes.io/projected/d9682c7b-f2d6-47b1-b9ec-39865c26687f-kube-api-access-ss6qt\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.549464 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.549379    2571 generic.go:356] "Generic (PLEG): container finished" podID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" containerID="6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b" exitCode=0
Sep 08 16:44:17.549464 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.549427    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"
Sep 08 16:44:17.549638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.549468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" event={"ID":"d9682c7b-f2d6-47b1-b9ec-39865c26687f","Type":"ContainerDied","Data":"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"}
Sep 08 16:44:17.549638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.549532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg" event={"ID":"d9682c7b-f2d6-47b1-b9ec-39865c26687f","Type":"ContainerDied","Data":"91a468acea0d7bc32eff0e6452bb7cbda279a3a9ff0c504cee729a36b0bd198f"}
Sep 08 16:44:17.549638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.549566    2571 scope.go:117] "RemoveContainer" containerID="6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"
Sep 08 16:44:17.559985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.559968    2571 scope.go:117] "RemoveContainer" containerID="6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"
Sep 08 16:44:17.560259 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:44:17.560239    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b\": container with ID starting with 6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b not found: ID does not exist" containerID="6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"
Sep 08 16:44:17.560307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.560269    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b"} err="failed to get container status \"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b\": rpc error: code = NotFound desc = could not find container \"6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b\": container with ID starting with 6d4e3efea0f70cf230f05c3205c2ccef55adcbd50e9e70dc5590bfd5dbcf2b8b not found: ID does not exist"
Sep 08 16:44:17.571431 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.571409    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:44:17.576612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.576592    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8dca27-6d7785d6b9-g2qsg"]
Sep 08 16:44:17.715034 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:17.714996    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:18.888384 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:18.888354    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" path="/var/lib/kubelet/pods/d9682c7b-f2d6-47b1-b9ec-39865c26687f/volumes"
Sep 08 16:44:21.195279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:21.195247    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:44:23.620461 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:23.620428    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:23.721109 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:23.721078    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:23.873950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:23.873845    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:25.224009 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:25.223951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:27.615017 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:27.614972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:40.179785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:40.179737    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:44:42.940380 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:42.940341    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:42.963902 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:42.963820    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:44:42.964120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:42.964089    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" podUID="d3375a70-d509-4256-9f3c-050a498a17fd" containerName="payload-processing" containerID="cri-o://0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c" gracePeriod=30
Sep 08 16:44:43.037572 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.037537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:43.142143 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.142097    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:43.224486 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.224460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:44:43.236475 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.236413    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:43.311828 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.311783    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume\") pod \"d3375a70-d509-4256-9f3c-050a498a17fd\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") "
Sep 08 16:44:43.312024 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.311899    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b478q\" (UniqueName: \"kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q\") pod \"d3375a70-d509-4256-9f3c-050a498a17fd\" (UID: \"d3375a70-d509-4256-9f3c-050a498a17fd\") "
Sep 08 16:44:43.312127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.312099    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3375a70-d509-4256-9f3c-050a498a17fd" (UID: "d3375a70-d509-4256-9f3c-050a498a17fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:43.312280 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.312262    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3375a70-d509-4256-9f3c-050a498a17fd-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:44:43.314376 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.314342    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q" (OuterVolumeSpecName: "kube-api-access-b478q") pod "d3375a70-d509-4256-9f3c-050a498a17fd" (UID: "d3375a70-d509-4256-9f3c-050a498a17fd"). InnerVolumeSpecName "kube-api-access-b478q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:43.413014 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.412983    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b478q\" (UniqueName: \"kubernetes.io/projected/d3375a70-d509-4256-9f3c-050a498a17fd-kube-api-access-b478q\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:44:43.566971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.566932    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:43.678515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.678485    2571 generic.go:356] "Generic (PLEG): container finished" podID="d3375a70-d509-4256-9f3c-050a498a17fd" containerID="0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c" exitCode=0
Sep 08 16:44:43.678689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.678524    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" event={"ID":"d3375a70-d509-4256-9f3c-050a498a17fd","Type":"ContainerDied","Data":"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"}
Sep 08 16:44:43.678689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.678562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9" event={"ID":"d3375a70-d509-4256-9f3c-050a498a17fd","Type":"ContainerDied","Data":"988a0e20f4d1c6dc650a31ffc16b231f81c86cb5108da606ede3af8ebf2b3275"}
Sep 08 16:44:43.678689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.678584    2571 scope.go:117] "RemoveContainer" containerID="0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"
Sep 08 16:44:43.678689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.678527    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"
Sep 08 16:44:43.687874 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.687856    2571 scope.go:117] "RemoveContainer" containerID="0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"
Sep 08 16:44:43.688154 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:44:43.688136    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c\": container with ID starting with 0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c not found: ID does not exist" containerID="0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"
Sep 08 16:44:43.688215 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.688161    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c"} err="failed to get container status \"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c\": rpc error: code = NotFound desc = could not find container \"0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c\": container with ID starting with 0957080b51f3af75cecbf559b7d51bcf1c534823bf3024364ee8e41d05cdb03c not found: ID does not exist"
Sep 08 16:44:43.700818 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.700789    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:44:43.704061 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:43.704038    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-352142-7d6dcbc87-dj2m9"]
Sep 08 16:44:44.887612 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:44.887567    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3375a70-d509-4256-9f3c-050a498a17fd" path="/var/lib/kubelet/pods/d3375a70-d509-4256-9f3c-050a498a17fd/volumes"
Sep 08 16:44:48.528321 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:48.528284    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:44:48.584260 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:48.584217    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:44:52.086922 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:52.086878    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:44:55.631927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:44:55.631888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:03.548436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.548390    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" podUID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" containerName="main" containerID="cri-o://3b8dff7467641ee92e5508c346e11940e088cdb0874dd9d1035ffd5a70e2abe9" gracePeriod=45
Sep 08 16:45:03.776564 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.776535    2571 generic.go:356] "Generic (PLEG): container finished" podID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" containerID="3b8dff7467641ee92e5508c346e11940e088cdb0874dd9d1035ffd5a70e2abe9" exitCode=0
Sep 08 16:45:03.776695 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.776602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" event={"ID":"c185d759-e6b4-471d-9a3f-4fbe0c1aa295","Type":"ContainerDied","Data":"3b8dff7467641ee92e5508c346e11940e088cdb0874dd9d1035ffd5a70e2abe9"}
Sep 08 16:45:03.794168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.794146    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:45:03.894985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.894956    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895001    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895044    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9njgk\" (UniqueName: \"kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895107    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895187 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs\") pod \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\" (UID: \"c185d759-e6b4-471d-9a3f-4fbe0c1aa295\") "
Sep 08 16:45:03.895441 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895295    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home" (OuterVolumeSpecName: "home") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:03.895441 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895335    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:03.895523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895498    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:03.895523 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895519    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:03.895629 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.895574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache" (OuterVolumeSpecName: "model-cache") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:03.897424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.897385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:03.897515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.897428    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk" (OuterVolumeSpecName: "kube-api-access-9njgk") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "kube-api-access-9njgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:03.897515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.897463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm" (OuterVolumeSpecName: "dshm") pod "c185d759-e6b4-471d-9a3f-4fbe0c1aa295" (UID: "c185d759-e6b4-471d-9a3f-4fbe0c1aa295"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:03.996870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.996836    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:03.996870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.996865    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:03.996870 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.996877    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:03.997092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:03.996888    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njgk\" (UniqueName: \"kubernetes.io/projected/c185d759-e6b4-471d-9a3f-4fbe0c1aa295-kube-api-access-9njgk\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:45:04.782474 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.782442    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"
Sep 08 16:45:04.783023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.782489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz" event={"ID":"c185d759-e6b4-471d-9a3f-4fbe0c1aa295","Type":"ContainerDied","Data":"27f55bd411333d65650dcff44e1c5c1397c88a8755565e84b3ca759c53d3dbcc"}
Sep 08 16:45:04.783023 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.782544    2571 scope.go:117] "RemoveContainer" containerID="3b8dff7467641ee92e5508c346e11940e088cdb0874dd9d1035ffd5a70e2abe9"
Sep 08 16:45:04.803909 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.803881    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:45:04.805591 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.805570    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-352142/ipp-route-352142-kserve-77f6b56ccf-htqfz"]
Sep 08 16:45:04.888110 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:04.888079    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" path="/var/lib/kubelet/pods/c185d759-e6b4-471d-9a3f-4fbe0c1aa295/volumes"
Sep 08 16:45:24.469971 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:24.469929    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:24.894831 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:24.894797    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:25.833724 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:25.833686    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:31.549604 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:31.549560    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:37.382574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:37.382535    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:37.480022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:37.479987    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:37.673835 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:37.673731    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:38.249556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:38.249525    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:48.504979 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:48.504874    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:48.603856 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:48.603820    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:48.704092 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:48.704067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:48.804785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:48.804663    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:49.091696 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:49.091667    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:51.578556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:51.578522    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:45:54.997315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:54.997276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:55.097423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:55.097375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:55.239535 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:55.239501    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:45:55.239721 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:55.239694    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy" containerID="cri-o://9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b" gracePeriod=30
Sep 08 16:45:55.286702 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:55.286612    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:45:55.719985 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:45:55.719944    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:00.489791 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.489742    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:46:00.577310 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577278    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577333    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577355    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577504    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577558    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t875m\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577583    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data\") pod \"382116d9-6863-4fb6-b2f7-423891198dd8\" (UID: \"382116d9-6863-4fb6-b2f7-423891198dd8\") "
Sep 08 16:46:00.577842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577687    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:00.577842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577737    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:00.578093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.577942    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:00.578093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.578014    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.578093 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.578037    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.578190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.578095    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:00.578305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.578262    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:00.579716 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.579686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:00.579955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.579933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m" (OuterVolumeSpecName: "kube-api-access-t875m") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "kube-api-access-t875m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:00.580144 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.580126    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:46:00.580222 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.580209    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "382116d9-6863-4fb6-b2f7-423891198dd8" (UID: "382116d9-6863-4fb6-b2f7-423891198dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:00.680583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680549    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/382116d9-6863-4fb6-b2f7-423891198dd8-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680576    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680585    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680595    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t875m\" (UniqueName: \"kubernetes.io/projected/382116d9-6863-4fb6-b2f7-423891198dd8-kube-api-access-t875m\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680604    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/382116d9-6863-4fb6-b2f7-423891198dd8-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680612    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:00.680881 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:00.680619    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/382116d9-6863-4fb6-b2f7-423891198dd8-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:01.043528 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.043493    2571 generic.go:356] "Generic (PLEG): container finished" podID="382116d9-6863-4fb6-b2f7-423891198dd8" containerID="9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b" exitCode=0
Sep 08 16:46:01.043697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.043564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerDied","Data":"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"}
Sep 08 16:46:01.043697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.043610    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg" event={"ID":"382116d9-6863-4fb6-b2f7-423891198dd8","Type":"ContainerDied","Data":"5c4e39d792ca92fb6c85ede471c0d80ff3b02e3093cfed90ec7b47d52b53cc19"}
Sep 08 16:46:01.043697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.043631    2571 scope.go:117] "RemoveContainer" containerID="9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"
Sep 08 16:46:01.043697 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.043565    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"
Sep 08 16:46:01.052789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.052756    2571 scope.go:117] "RemoveContainer" containerID="55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"
Sep 08 16:46:01.060644 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.060604    2571 scope.go:117] "RemoveContainer" containerID="9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"
Sep 08 16:46:01.061160 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:01.060971    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b\": container with ID starting with 9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b not found: ID does not exist" containerID="9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"
Sep 08 16:46:01.061160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.061010    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b"} err="failed to get container status \"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b\": rpc error: code = NotFound desc = could not find container \"9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b\": container with ID starting with 9c67f9a684b58596976ca5cb080b457a3b02c4e80a22a43c130621d1d5ba4f8b not found: ID does not exist"
Sep 08 16:46:01.061160 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.061033    2571 scope.go:117] "RemoveContainer" containerID="55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"
Sep 08 16:46:01.061350 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:01.061311    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c\": container with ID starting with 55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c not found: ID does not exist" containerID="55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"
Sep 08 16:46:01.061350 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.061339    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c"} err="failed to get container status \"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c\": rpc error: code = NotFound desc = could not find container \"55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c\": container with ID starting with 55cd8cb8f6375c4655a76d48d6fee7e34d01d5c08381e2a3f9b3d47543d2d73c not found: ID does not exist"
Sep 08 16:46:01.062760 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.062741    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:46:01.068556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:01.068535    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-995b22-openshift-default-797848d5f4-q25vg"]
Sep 08 16:46:02.887842 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:02.887805    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" path="/var/lib/kubelet/pods/382116d9-6863-4fb6-b2f7-423891198dd8/volumes"
Sep 08 16:46:06.200207 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.200173    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:46:06.200606 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.200435    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" podUID="4242e669-9d37-48db-88cc-aa30436f84b8" containerName="maas-api" containerID="cri-o://02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8" gracePeriod=30
Sep 08 16:46:06.227988 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.227959    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:06.247429 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.247386    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:46:06.247711 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.247652    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" podUID="7d7a8930-c090-499d-9431-9794be45cd63" containerName="payload-processing" containerID="cri-o://e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205" gracePeriod=30
Sep 08 16:46:06.329600 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.329562    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:06.424861 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.424818    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:06.489927 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.489879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:06.502132 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.502102    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:46:06.530315 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.530272    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca\") pod \"4242e669-9d37-48db-88cc-aa30436f84b8\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") "
Sep 08 16:46:06.530488 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.530368    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") pod \"4242e669-9d37-48db-88cc-aa30436f84b8\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") "
Sep 08 16:46:06.530705 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.530668    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4242e669-9d37-48db-88cc-aa30436f84b8" (UID: "4242e669-9d37-48db-88cc-aa30436f84b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:06.530828 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.530399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs\") pod \"4242e669-9d37-48db-88cc-aa30436f84b8\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") "
Sep 08 16:46:06.530917 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.530899    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b88hb\" (UniqueName: \"kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb\") pod \"4242e669-9d37-48db-88cc-aa30436f84b8\" (UID: \"4242e669-9d37-48db-88cc-aa30436f84b8\") "
Sep 08 16:46:06.531245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.531223    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4242e669-9d37-48db-88cc-aa30436f84b8-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.533004 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.532976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4242e669-9d37-48db-88cc-aa30436f84b8" (UID: "4242e669-9d37-48db-88cc-aa30436f84b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:46:06.533176 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.533154    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4242e669-9d37-48db-88cc-aa30436f84b8" (UID: "4242e669-9d37-48db-88cc-aa30436f84b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:46:06.533392 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.533369    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb" (OuterVolumeSpecName: "kube-api-access-b88hb") pod "4242e669-9d37-48db-88cc-aa30436f84b8" (UID: "4242e669-9d37-48db-88cc-aa30436f84b8"). InnerVolumeSpecName "kube-api-access-b88hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:06.535112 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.535097    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:46:06.632452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632416    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume\") pod \"7d7a8930-c090-499d-9431-9794be45cd63\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") "
Sep 08 16:46:06.632452 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632455    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4mz5\" (UniqueName: \"kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5\") pod \"7d7a8930-c090-499d-9431-9794be45cd63\" (UID: \"7d7a8930-c090-499d-9431-9794be45cd63\") "
Sep 08 16:46:06.632798 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632761    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88hb\" (UniqueName: \"kubernetes.io/projected/4242e669-9d37-48db-88cc-aa30436f84b8-kube-api-access-b88hb\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.632858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632806    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.632858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632821    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4242e669-9d37-48db-88cc-aa30436f84b8-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.632858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.632836    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d7a8930-c090-499d-9431-9794be45cd63" (UID: "7d7a8930-c090-499d-9431-9794be45cd63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:06.634789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.634743    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5" (OuterVolumeSpecName: "kube-api-access-l4mz5") pod "7d7a8930-c090-499d-9431-9794be45cd63" (UID: "7d7a8930-c090-499d-9431-9794be45cd63"). InnerVolumeSpecName "kube-api-access-l4mz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:06.733613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.733512    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7a8930-c090-499d-9431-9794be45cd63-plugins-config-volume\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.733613 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.733547    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mz5\" (UniqueName: \"kubernetes.io/projected/7d7a8930-c090-499d-9431-9794be45cd63-kube-api-access-l4mz5\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.816347 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:06.816312    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:07.073290 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.073259    2571 generic.go:356] "Generic (PLEG): container finished" podID="4242e669-9d37-48db-88cc-aa30436f84b8" containerID="02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8" exitCode=0
Sep 08 16:46:07.073489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.073330    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"
Sep 08 16:46:07.073489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.073339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" event={"ID":"4242e669-9d37-48db-88cc-aa30436f84b8","Type":"ContainerDied","Data":"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"}
Sep 08 16:46:07.073489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.073399    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4" event={"ID":"4242e669-9d37-48db-88cc-aa30436f84b8","Type":"ContainerDied","Data":"07140653c9eea385d13a76396dc3f396143ecd19c4ebe753003ef57ceb7a5031"}
Sep 08 16:46:07.073489 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.073422    2571 scope.go:117] "RemoveContainer" containerID="02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"
Sep 08 16:46:07.074932 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.074911    2571 generic.go:356] "Generic (PLEG): container finished" podID="7d7a8930-c090-499d-9431-9794be45cd63" containerID="e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205" exitCode=0
Sep 08 16:46:07.075056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.074949    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" event={"ID":"7d7a8930-c090-499d-9431-9794be45cd63","Type":"ContainerDied","Data":"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"}
Sep 08 16:46:07.075056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.074980    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"
Sep 08 16:46:07.075056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.074987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z" event={"ID":"7d7a8930-c090-499d-9431-9794be45cd63","Type":"ContainerDied","Data":"8eb837da396613667dc33db46c4df82ec4f91a062030f14505081c79389c584b"}
Sep 08 16:46:07.082365 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.082350    2571 scope.go:117] "RemoveContainer" containerID="02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"
Sep 08 16:46:07.082683 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:07.082655    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8\": container with ID starting with 02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8 not found: ID does not exist" containerID="02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"
Sep 08 16:46:07.082786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.082694    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8"} err="failed to get container status \"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8\": rpc error: code = NotFound desc = could not find container \"02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8\": container with ID starting with 02cb51391a0b56a18f2791688321d4d39496710ff640aeaa6874efdb73003ac8 not found: ID does not exist"
Sep 08 16:46:07.082786 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.082717    2571 scope.go:117] "RemoveContainer" containerID="e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"
Sep 08 16:46:07.090283 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.090199    2571 scope.go:117] "RemoveContainer" containerID="e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"
Sep 08 16:46:07.090891 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:07.090865    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205\": container with ID starting with e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205 not found: ID does not exist" containerID="e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"
Sep 08 16:46:07.090995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.090897    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205"} err="failed to get container status \"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205\": rpc error: code = NotFound desc = could not find container \"e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205\": container with ID starting with e11a75b7e99fe7b917582740f4d62ea50d94dc1ef7a0ca64cddd9b8346e18205 not found: ID does not exist"
Sep 08 16:46:07.092056 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.092037    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:46:07.096240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.096218    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bce20b-6559656df5-f8ld4"]
Sep 08 16:46:07.107795 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.107750    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:46:07.110010 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:07.109990    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bce20b-55ff49bdf5-r476z"]
Sep 08 16:46:08.877241 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:08.877203    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:46:08.889031 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:08.889001    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4242e669-9d37-48db-88cc-aa30436f84b8" path="/var/lib/kubelet/pods/4242e669-9d37-48db-88cc-aa30436f84b8/volumes"
Sep 08 16:46:08.889342 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:08.889330    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7a8930-c090-499d-9431-9794be45cd63" path="/var/lib/kubelet/pods/7d7a8930-c090-499d-9431-9794be45cd63/volumes"
Sep 08 16:46:12.919556 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:12.919517    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:13.170228 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:13.170134    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:46:13.170398 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:13.170349    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy" containerID="cri-o://26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1" gracePeriod=30
Sep 08 16:46:13.713515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:13.713481    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:13.814961 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:13.814928    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:13.949292 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:13.949260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:14.050871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:14.050785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:18.425101 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.425075    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:46:18.542780 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542704    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542780 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542791    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542817    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542861    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542886    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.542977 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542952    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lc2t\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.543236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542966    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:18.543236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.542982    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.543236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543089    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs\") pod \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\" (UID: \"09999f74-5a99-4c6e-9d21-a3e7f0070ce5\") "
Sep 08 16:46:18.543236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543184    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data" (OuterVolumeSpecName: "istio-data") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:18.543236 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543204    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:18.543468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:18.543468 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543447    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:18.543557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543519    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.543557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543539    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istiod-ca-cert\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.543557 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543553    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-data\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.543681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.543566    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-credential-socket\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.545273 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.545250    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:46:18.545366 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.545347    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t" (OuterVolumeSpecName: "kube-api-access-9lc2t") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "kube-api-access-9lc2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:18.545540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.545514    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:46:18.545540 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.545523    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token" (OuterVolumeSpecName: "istio-token") pod "09999f74-5a99-4c6e-9d21-a3e7f0070ce5" (UID: "09999f74-5a99-4c6e-9d21-a3e7f0070ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:18.644783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.644731    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lc2t\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-kube-api-access-9lc2t\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.644783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.644763    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-token\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.644783 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.644795    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-workload-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.644975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.644803    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-podinfo\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:18.644975 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:18.644811    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09999f74-5a99-4c6e-9d21-a3e7f0070ce5-istio-envoy\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:19.134053 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.134024    2571 generic.go:356] "Generic (PLEG): container finished" podID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerID="26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1" exitCode=0
Sep 08 16:46:19.134200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.134071    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"
Sep 08 16:46:19.134200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.134076    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerDied","Data":"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"}
Sep 08 16:46:19.134200 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.134180    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc" event={"ID":"09999f74-5a99-4c6e-9d21-a3e7f0070ce5","Type":"ContainerDied","Data":"80b122b9dfd16cb06b68c27677a05b64215b04cb802d11242b807191cc1cdb72"}
Sep 08 16:46:19.134307 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.134202    2571 scope.go:117] "RemoveContainer" containerID="26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"
Sep 08 16:46:19.142874 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.142829    2571 scope.go:117] "RemoveContainer" containerID="c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"
Sep 08 16:46:19.150354 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.150333    2571 scope.go:117] "RemoveContainer" containerID="26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"
Sep 08 16:46:19.150600 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:19.150582    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1\": container with ID starting with 26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1 not found: ID does not exist" containerID="26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"
Sep 08 16:46:19.150651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.150607    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1"} err="failed to get container status \"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1\": rpc error: code = NotFound desc = could not find container \"26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1\": container with ID starting with 26be5a9222ea85873ac54c2f2f6dd646e4888d9cd120c590e32bbf185a6c27a1 not found: ID does not exist"
Sep 08 16:46:19.150651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.150626    2571 scope.go:117] "RemoveContainer" containerID="c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"
Sep 08 16:46:19.150913 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:46:19.150897    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251\": container with ID starting with c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251 not found: ID does not exist" containerID="c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"
Sep 08 16:46:19.150969 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.150916    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251"} err="failed to get container status \"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251\": rpc error: code = NotFound desc = could not find container \"c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251\": container with ID starting with c4bb76d527907435f4abbd029fe5df4d0073bb4ef1770a33fdaa63878c9af251 not found: ID does not exist"
Sep 08 16:46:19.157457 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.157433    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:46:19.167434 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:19.167412    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bce20b-openshift-default-b78cdc88d-xbxmc"]
Sep 08 16:46:20.887519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:20.887490    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" path="/var/lib/kubelet/pods/09999f74-5a99-4c6e-9d21-a3e7f0070ce5/volumes"
Sep 08 16:46:21.888641 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:21.888589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:23.971116 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:23.971079    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:46:23.971601 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:23.971323    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" podUID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" containerName="maas-api" containerID="cri-o://8ef8f8f16755418460d54043349e91704c10329f83c00cfd2f2d65a96a6a34c7" gracePeriod=30
Sep 08 16:46:24.006050 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.006004    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:24.080247 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.080214    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:24.151990 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.151958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:24.161190 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.161157    2571 generic.go:356] "Generic (PLEG): container finished" podID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" containerID="8ef8f8f16755418460d54043349e91704c10329f83c00cfd2f2d65a96a6a34c7" exitCode=0
Sep 08 16:46:24.161345 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.161223    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" event={"ID":"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f","Type":"ContainerDied","Data":"8ef8f8f16755418460d54043349e91704c10329f83c00cfd2f2d65a96a6a34c7"}
Sep 08 16:46:24.229948 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.229886    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:46:24.285046 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.285007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:24.297296 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.297267    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") pod \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") "
Sep 08 16:46:24.297436 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.297323    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkwg6\" (UniqueName: \"kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6\") pod \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") "
Sep 08 16:46:24.297476 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.297438    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs\") pod \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") "
Sep 08 16:46:24.297520 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.297472    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca\") pod \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\" (UID: \"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f\") "
Sep 08 16:46:24.297896 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.297867    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" (UID: "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:24.299565 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.299541    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" (UID: "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:46:24.299645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.299571    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" (UID: "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:46:24.299645 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.299603    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6" (OuterVolumeSpecName: "kube-api-access-zkwg6") pod "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" (UID: "8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f"). InnerVolumeSpecName "kube-api-access-zkwg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:24.346789 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.346734    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:24.398370 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.398336    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.398370 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.398371    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkwg6\" (UniqueName: \"kubernetes.io/projected/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-kube-api-access-zkwg6\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.398583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.398384    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.398583 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.398392    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.587568 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:24.587537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:25.167204 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:25.167123    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"
Sep 08 16:46:25.167668 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:25.167130    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf" event={"ID":"8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f","Type":"ContainerDied","Data":"3e25eb80ef3c5d37efbb15bbc14a3898337450de41fc4188e931323faa74409f"}
Sep 08 16:46:25.167668 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:25.167259    2571 scope.go:117] "RemoveContainer" containerID="8ef8f8f16755418460d54043349e91704c10329f83c00cfd2f2d65a96a6a34c7"
Sep 08 16:46:25.183671 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:25.183646    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:46:25.191178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:25.191155    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e22de8-57df8b798b-mmppf"]
Sep 08 16:46:26.480443 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:26.480399    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:46:26.887254 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:26.887222    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" path="/var/lib/kubelet/pods/8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f/volumes"
Sep 08 16:46:30.549815 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:30.549760    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:30.648127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:30.648094    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:30.816886 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:30.816852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:46:35.000582 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:46:35.000542    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:47:50.970675 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:50.970642    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:47:50.975469 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:50.975447    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lp7kv_1c20d2ca-5bf2-4624-bb7f-5cd35eb4a9f5/ovn-acl-logging/0.log"
Sep 08 16:47:53.648004 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:53.647974    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:47:54.572978 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:54.572943    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:47:55.160284 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:55.160223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:47:56.289589 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:47:56.289558    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:48:18.181757 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:48:18.181721    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:48:25.971521 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:48:25.971487    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:48:32.672912 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:48:32.672874    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:48:49.177814 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:48:49.177762    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:49:24.657681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:49:24.657631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:49:24.722949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:49:24.722905    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:50:15.977753 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:15.977667    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:50:23.874689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:23.874650    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:50:29.475934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:29.475898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:50:31.371448 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371412    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371746    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7a8930-c090-499d-9431-9794be45cd63" containerName="payload-processing"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371759    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7a8930-c090-499d-9431-9794be45cd63" containerName="payload-processing"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371792    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4242e669-9d37-48db-88cc-aa30436f84b8" containerName="maas-api"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371798    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4242e669-9d37-48db-88cc-aa30436f84b8" containerName="maas-api"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371804    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371809    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371820    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371829    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371838    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3375a70-d509-4256-9f3c-050a498a17fd" containerName="payload-processing"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371843    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3375a70-d509-4256-9f3c-050a498a17fd" containerName="payload-processing"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371856    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" containerName="main"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371861    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" containerName="main"
Sep 08 16:50:31.371862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371868    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" containerName="maas-api"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371874    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" containerName="maas-api"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371880    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" containerName="payload-processing"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371885    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" containerName="payload-processing"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371895    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371904    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371914    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371918    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371970    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371983    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="382116d9-6863-4fb6-b2f7-423891198dd8" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371991    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.371997    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4242e669-9d37-48db-88cc-aa30436f84b8" containerName="maas-api"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372002    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="09999f74-5a99-4c6e-9d21-a3e7f0070ce5" containerName="istio-proxy"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372009    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cdd3173-dc7f-4b31-8487-2eb13f2d5c7f" containerName="maas-api"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372014    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7a8930-c090-499d-9431-9794be45cd63" containerName="payload-processing"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372019    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3375a70-d509-4256-9f3c-050a498a17fd" containerName="payload-processing"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372024    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9682c7b-f2d6-47b1-b9ec-39865c26687f" containerName="payload-processing"
Sep 08 16:50:31.372326 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.372030    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c185d759-e6b4-471d-9a3f-4fbe0c1aa295" containerName="main"
Sep 08 16:50:31.375128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.375110    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.377462 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.377440    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2f4cd352-kserve-self-signed-certs\""
Sep 08 16:50:31.384903 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.384881    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:50:31.399862 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.399830    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:50:31.438698 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxt7\" (UniqueName: \"kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.438847 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.438847 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.438847 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.439011 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.439011 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.438938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.539904 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.539872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.539944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.539978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxt7\" (UniqueName: \"kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540095 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540395 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540483 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540536 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.540666 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.540645    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.542827 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.542805    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.542955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.542876    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.547824 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.547757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxt7\" (UniqueName: \"kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7\") pod \"e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") " pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.685454 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.685378    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:31.956424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:31.956320    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:50:31.959060 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:50:31.959036    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3218fa_a172_485b_8b68_040920653f93.slice/crio-810f82d0f68a26bc248824746a4d0d25dd228d3bc93fe76e06ab70d59e30c035 WatchSource:0}: Error finding container 810f82d0f68a26bc248824746a4d0d25dd228d3bc93fe76e06ab70d59e30c035: Status 404 returned error can't find the container with id 810f82d0f68a26bc248824746a4d0d25dd228d3bc93fe76e06ab70d59e30c035
Sep 08 16:50:32.270389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:32.270293    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" event={"ID":"eb3218fa-a172-485b-8b68-040920653f93","Type":"ContainerStarted","Data":"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"}
Sep 08 16:50:32.270389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:32.270341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" event={"ID":"eb3218fa-a172-485b-8b68-040920653f93","Type":"ContainerStarted","Data":"810f82d0f68a26bc248824746a4d0d25dd228d3bc93fe76e06ab70d59e30c035"}
Sep 08 16:50:32.287141 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:32.287091    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" podStartSLOduration=1.287076438 podStartE2EDuration="1.287076438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:32.286827942 +0000 UTC m=+2261.930904634" watchObservedRunningTime="2026-09-08 16:50:32.287076438 +0000 UTC m=+2261.931153130"
Sep 08 16:50:37.993381 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:37.993344    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:50:41.686252 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:41.686217    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:41.686651 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:41.686306    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:41.698532 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:41.698505    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:42.326207 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:42.326179    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:50:48.892013 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.891978    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:50:48.895694 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.895673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.898499 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.898481    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2f4cd352-kserve-self-signed-certs\""
Sep 08 16:50:48.904003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.903983    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:50:48.990100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbt6\" (UniqueName: \"kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.990100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.990100 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.990263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990154    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.990263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:48.990263 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:48.990242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091654 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbt6\" (UniqueName: \"kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.091955 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.091885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.092120 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.092094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.092183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.092106    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.092183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.092161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.094240 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.094218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.094491 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.094473    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.099762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.099739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbt6\" (UniqueName: \"kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6\") pod \"e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") " pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.206424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.206393    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:49.437646 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.437596    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:50:49.476950 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:49.476916    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:50:49.480055 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:50:49.480032    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc05ddfc5_42b0_4eb6_bb88_597ae86c4671.slice/crio-fe9ba446460c612002a863232b90a3af97d727c448c5b98487e7a2e1d81b10b3 WatchSource:0}: Error finding container fe9ba446460c612002a863232b90a3af97d727c448c5b98487e7a2e1d81b10b3: Status 404 returned error can't find the container with id fe9ba446460c612002a863232b90a3af97d727c448c5b98487e7a2e1d81b10b3
Sep 08 16:50:50.352672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:50.352628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" event={"ID":"c05ddfc5-42b0-4eb6-bb88-597ae86c4671","Type":"ContainerStarted","Data":"24172155577de73d1465463dbbca91d1e20cd15a6ef301ef36453922865d324a"}
Sep 08 16:50:50.352672 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:50.352677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" event={"ID":"c05ddfc5-42b0-4eb6-bb88-597ae86c4671","Type":"ContainerStarted","Data":"fe9ba446460c612002a863232b90a3af97d727c448c5b98487e7a2e1d81b10b3"}
Sep 08 16:50:50.372968 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:50.372916    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" podStartSLOduration=2.37290113 podStartE2EDuration="2.37290113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:50.371251476 +0000 UTC m=+2280.015328176" watchObservedRunningTime="2026-09-08 16:50:50.37290113 +0000 UTC m=+2280.016977822"
Sep 08 16:50:51.588071 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:51.588038    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:50:59.206837 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:59.206794    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:59.207229 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:59.206857    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:59.219183 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:59.219156    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:50:59.407747 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:50:59.407716    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:51:03.913246 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:03.913215    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:51:03.946822 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:03.946784    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:51:05.343924 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:05.343889    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:51:05.356871 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:05.356841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:51:05.776051 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:05.775969    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:13.285876 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:13.285839    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:18.776286 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:18.776251    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:18.932689 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:18.932627    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" podUID="eb3218fa-a172-485b-8b68-040920653f93" containerName="main" containerID="cri-o://107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6" gracePeriod=45
Sep 08 16:51:19.176825 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.176802    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:51:19.244032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244004    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244050    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244092    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244165 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244127    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcxt7\" (UniqueName: \"kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244226    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244256    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm\") pod \"eb3218fa-a172-485b-8b68-040920653f93\" (UID: \"eb3218fa-a172-485b-8b68-040920653f93\") "
Sep 08 16:51:19.244325 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244298    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home" (OuterVolumeSpecName: "home") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:19.244463 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244350    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:19.244463 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244377    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:19.244640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244620    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.244640 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244638    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.245012 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.244647    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.246447 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.246424    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm" (OuterVolumeSpecName: "dshm") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:19.246530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.246478    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7" (OuterVolumeSpecName: "kube-api-access-gcxt7") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "kube-api-access-gcxt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:19.246530 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.246505    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb3218fa-a172-485b-8b68-040920653f93" (UID: "eb3218fa-a172-485b-8b68-040920653f93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:19.345993 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.345971    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxt7\" (UniqueName: \"kubernetes.io/projected/eb3218fa-a172-485b-8b68-040920653f93-kube-api-access-gcxt7\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.345993 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.345991    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3218fa-a172-485b-8b68-040920653f93-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.346117 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.346001    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3218fa-a172-485b-8b68-040920653f93-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:19.488713 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.488680    2571 generic.go:356] "Generic (PLEG): container finished" podID="eb3218fa-a172-485b-8b68-040920653f93" containerID="107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6" exitCode=0
Sep 08 16:51:19.488928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.488727    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"
Sep 08 16:51:19.488928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.488758    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" event={"ID":"eb3218fa-a172-485b-8b68-040920653f93","Type":"ContainerDied","Data":"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"}
Sep 08 16:51:19.488928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.488823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9" event={"ID":"eb3218fa-a172-485b-8b68-040920653f93","Type":"ContainerDied","Data":"810f82d0f68a26bc248824746a4d0d25dd228d3bc93fe76e06ab70d59e30c035"}
Sep 08 16:51:19.488928 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.488843    2571 scope.go:117] "RemoveContainer" containerID="107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"
Sep 08 16:51:19.497272 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.497253    2571 scope.go:117] "RemoveContainer" containerID="107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"
Sep 08 16:51:19.497537 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:51:19.497521    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6\": container with ID starting with 107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6 not found: ID does not exist" containerID="107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"
Sep 08 16:51:19.497610 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.497544    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6"} err="failed to get container status \"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6\": rpc error: code = NotFound desc = could not find container \"107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6\": container with ID starting with 107af47f27104496fee6bbf8a24d957c8d3006519f7092d546b637b0ca51ebc6 not found: ID does not exist"
Sep 08 16:51:19.511787 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.511756    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:51:19.515802 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:19.515754    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2f4cd352-kserve-7d79854959-ng8p9"]
Sep 08 16:51:20.363740 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.363693    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" podUID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" containerName="main" containerID="cri-o://24172155577de73d1465463dbbca91d1e20cd15a6ef301ef36453922865d324a" gracePeriod=45
Sep 08 16:51:20.495841 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.495810    2571 generic.go:356] "Generic (PLEG): container finished" podID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" containerID="24172155577de73d1465463dbbca91d1e20cd15a6ef301ef36453922865d324a" exitCode=0
Sep 08 16:51:20.495956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.495868    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" event={"ID":"c05ddfc5-42b0-4eb6-bb88-597ae86c4671","Type":"ContainerDied","Data":"24172155577de73d1465463dbbca91d1e20cd15a6ef301ef36453922865d324a"}
Sep 08 16:51:20.601327 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.601296    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:51:20.656679 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656609    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.656826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656683    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.656826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.656826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656739    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.656826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656796    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.657038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656830    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmbt6\" (UniqueName: \"kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6\") pod \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\" (UID: \"c05ddfc5-42b0-4eb6-bb88-597ae86c4671\") "
Sep 08 16:51:20.657038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.656910    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home" (OuterVolumeSpecName: "home") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:20.657038 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.657019    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache" (OuterVolumeSpecName: "model-cache") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:20.657210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.657113    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:20.657210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.657176    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-home\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.657210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.657190    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-model-cache\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.657210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.657199    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tmp-dir\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.659004 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.658982    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm" (OuterVolumeSpecName: "dshm") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:20.659099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.659005    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:20.659099 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.659041    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6" (OuterVolumeSpecName: "kube-api-access-fmbt6") pod "c05ddfc5-42b0-4eb6-bb88-597ae86c4671" (UID: "c05ddfc5-42b0-4eb6-bb88-597ae86c4671"). InnerVolumeSpecName "kube-api-access-fmbt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:20.757785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.757740    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-tls-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.757785 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.757784    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-dshm\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.757919 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.757798    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmbt6\" (UniqueName: \"kubernetes.io/projected/c05ddfc5-42b0-4eb6-bb88-597ae86c4671-kube-api-access-fmbt6\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:20.887429 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:20.887391    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3218fa-a172-485b-8b68-040920653f93" path="/var/lib/kubelet/pods/eb3218fa-a172-485b-8b68-040920653f93/volumes"
Sep 08 16:51:21.500761 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:21.500725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d" event={"ID":"c05ddfc5-42b0-4eb6-bb88-597ae86c4671","Type":"ContainerDied","Data":"fe9ba446460c612002a863232b90a3af97d727c448c5b98487e7a2e1d81b10b3"}
Sep 08 16:51:21.501235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:21.500798    2571 scope.go:117] "RemoveContainer" containerID="24172155577de73d1465463dbbca91d1e20cd15a6ef301ef36453922865d324a"
Sep 08 16:51:21.501235 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:21.500744    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"
Sep 08 16:51:21.517884 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:21.517854    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:51:21.522128 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:21.522105    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2f4cd352-kserve-57f646557f-s8m5d"]
Sep 08 16:51:22.887306 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:22.887276    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" path="/var/lib/kubelet/pods/c05ddfc5-42b0-4eb6-bb88-597ae86c4671/volumes"
Sep 08 16:51:26.072343 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.072310    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:26.395662 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.395581    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:26.395982 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.395968    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3218fa-a172-485b-8b68-040920653f93" containerName="main"
Sep 08 16:51:26.396032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.395985    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3218fa-a172-485b-8b68-040920653f93" containerName="main"
Sep 08 16:51:26.396032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.395999    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" containerName="main"
Sep 08 16:51:26.396032 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.396005    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" containerName="main"
Sep 08 16:51:26.396131 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.396079    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3218fa-a172-485b-8b68-040920653f93" containerName="main"
Sep 08 16:51:26.396131 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.396086    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05ddfc5-42b0-4eb6-bb88-597ae86c4671" containerName="main"
Sep 08 16:51:26.399006 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.398985    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.401626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.401606    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:51:26.402580 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.402563    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:51:26.402678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.402581    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksssl\""
Sep 08 16:51:26.402678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.402566    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:51:26.402678 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.402608    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 16:51:26.409548 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.409527    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:26.411663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.411643    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"]
Sep 08 16:51:26.414944 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.414927    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.427391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.427276    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"]
Sep 08 16:51:26.502643 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502614    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8rb\" (UniqueName: \"kubernetes.io/projected/8a6df7df-2857-485a-814a-9b369abda5e7-kube-api-access-cg8rb\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.502808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502667    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.502808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.502808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502731    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-metrics-certs\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.502808 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-maas-api-tls\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.502947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6df7df-2857-485a-814a-9b369abda5e7-openshift-service-ca\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.502947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58b8\" (UniqueName: \"kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.502947 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.502890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.604168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.604168 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.604332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-metrics-certs\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.604332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-maas-api-tls\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.604332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6df7df-2857-485a-814a-9b369abda5e7-openshift-service-ca\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.604332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58b8\" (UniqueName: \"kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.604332 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604302    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.604570 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.604358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8rb\" (UniqueName: \"kubernetes.io/projected/8a6df7df-2857-485a-814a-9b369abda5e7-kube-api-access-cg8rb\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.605087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.605055    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6df7df-2857-485a-814a-9b369abda5e7-openshift-service-ca\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.605198 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.605148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.606914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.606891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.606914 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.606904    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-maas-api-tls\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.607245 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.607227    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.607305 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.607229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6df7df-2857-485a-814a-9b369abda5e7-metrics-certs\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.611917 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.611896    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8rb\" (UniqueName: \"kubernetes.io/projected/8a6df7df-2857-485a-814a-9b369abda5e7-kube-api-access-cg8rb\") pod \"maas-api-7bbd6f9566-jszv5\" (UID: \"8a6df7df-2857-485a-814a-9b369abda5e7\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.612022 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.611993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58b8\" (UniqueName: \"kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8\") pod \"maas-api-5df4497bcd-vdzrr\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.709897 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.709827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:26.727820 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.727793    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:26.991792 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.991556    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:26.998186 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:51:26.998080    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda790194b_f439_4f21_9c85_88ac0fd96b80.slice/crio-2f501d3943907a575dd664659baceac2a80b0fb1e3ff1501ab05d981f30d14e2 WatchSource:0}: Error finding container 2f501d3943907a575dd664659baceac2a80b0fb1e3ff1501ab05d981f30d14e2: Status 404 returned error can't find the container with id 2f501d3943907a575dd664659baceac2a80b0fb1e3ff1501ab05d981f30d14e2
Sep 08 16:51:26.999761 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:26.999725    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:51:27.021858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:27.021796    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"]
Sep 08 16:51:27.025107 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:51:27.025068    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6df7df_2857_485a_814a_9b369abda5e7.slice/crio-0ab47cb37587d2c5cefff6a461c8768d98cca3c73fe39f13599aa0656a8c6f61 WatchSource:0}: Error finding container 0ab47cb37587d2c5cefff6a461c8768d98cca3c73fe39f13599aa0656a8c6f61: Status 404 returned error can't find the container with id 0ab47cb37587d2c5cefff6a461c8768d98cca3c73fe39f13599aa0656a8c6f61
Sep 08 16:51:27.529752 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:27.529711    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5" event={"ID":"8a6df7df-2857-485a-814a-9b369abda5e7","Type":"ContainerStarted","Data":"0ab47cb37587d2c5cefff6a461c8768d98cca3c73fe39f13599aa0656a8c6f61"}
Sep 08 16:51:27.530949 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:27.530920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" event={"ID":"a790194b-f439-4f21-9c85-88ac0fd96b80","Type":"ContainerStarted","Data":"2f501d3943907a575dd664659baceac2a80b0fb1e3ff1501ab05d981f30d14e2"}
Sep 08 16:51:28.536807 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.536747    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5" event={"ID":"8a6df7df-2857-485a-814a-9b369abda5e7","Type":"ContainerStarted","Data":"c637bfdbb7db7698aa56990c5339d15dd750e6c62182b49c22b4a7452d8625e3"}
Sep 08 16:51:28.537279 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.536835    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:28.538163 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.538141    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" event={"ID":"a790194b-f439-4f21-9c85-88ac0fd96b80","Type":"ContainerStarted","Data":"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"}
Sep 08 16:51:28.538280 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.538261    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:28.559826 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.559785    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5" podStartSLOduration=1.680649968 podStartE2EDuration="2.559752302s" totalImagesPullingTime="879.102334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:28.557079299 +0000 UTC m=+2318.201155989" watchObservedRunningTime="2026-09-08 16:51:28.559752302 +0000 UTC m=+2318.203829035"
Sep 08 16:51:28.577652 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:28.577604    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" podStartSLOduration=1.567651696 podStartE2EDuration="2.577591118s" totalImagesPullingTime="1.009939422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:28.57598457 +0000 UTC m=+2318.220061262" watchObservedRunningTime="2026-09-08 16:51:28.577591118 +0000 UTC m=+2318.221667809"
Sep 08 16:51:34.080444 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.080407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:34.548210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.548129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:34.548574 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.548551    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-jszv5"
Sep 08 16:51:34.628391 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.628357    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:34.628579 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.628543    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" podUID="a790194b-f439-4f21-9c85-88ac0fd96b80" containerName="maas-api" containerID="cri-o://a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601" gracePeriod=30
Sep 08 16:51:34.870941 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.870915    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:34.973335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973303    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls\") pod \"a790194b-f439-4f21-9c85-88ac0fd96b80\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") "
Sep 08 16:51:34.973483 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca\") pod \"a790194b-f439-4f21-9c85-88ac0fd96b80\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") "
Sep 08 16:51:34.973483 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973384    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs\") pod \"a790194b-f439-4f21-9c85-88ac0fd96b80\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") "
Sep 08 16:51:34.973579 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973497    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g58b8\" (UniqueName: \"kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8\") pod \"a790194b-f439-4f21-9c85-88ac0fd96b80\" (UID: \"a790194b-f439-4f21-9c85-88ac0fd96b80\") "
Sep 08 16:51:34.973741 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973717    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a790194b-f439-4f21-9c85-88ac0fd96b80" (UID: "a790194b-f439-4f21-9c85-88ac0fd96b80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:34.973863 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.973841    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a790194b-f439-4f21-9c85-88ac0fd96b80-openshift-service-ca\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:34.975681 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.975654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8" (OuterVolumeSpecName: "kube-api-access-g58b8") pod "a790194b-f439-4f21-9c85-88ac0fd96b80" (UID: "a790194b-f439-4f21-9c85-88ac0fd96b80"). InnerVolumeSpecName "kube-api-access-g58b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:34.976070 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.976045    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a790194b-f439-4f21-9c85-88ac0fd96b80" (UID: "a790194b-f439-4f21-9c85-88ac0fd96b80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:34.976156 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:34.976091    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a790194b-f439-4f21-9c85-88ac0fd96b80" (UID: "a790194b-f439-4f21-9c85-88ac0fd96b80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:35.074854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.074821    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-maas-api-tls\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:35.074854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.074844    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a790194b-f439-4f21-9c85-88ac0fd96b80-metrics-certs\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:35.074854 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.074855    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58b8\" (UniqueName: \"kubernetes.io/projected/a790194b-f439-4f21-9c85-88ac0fd96b80-kube-api-access-g58b8\") on node \"ip-10-0-135-163.ec2.internal\" DevicePath \"\""
Sep 08 16:51:35.572858 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.572815    2571 generic.go:356] "Generic (PLEG): container finished" podID="a790194b-f439-4f21-9c85-88ac0fd96b80" containerID="a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601" exitCode=0
Sep 08 16:51:35.573278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.572863    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"
Sep 08 16:51:35.573278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.572902    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" event={"ID":"a790194b-f439-4f21-9c85-88ac0fd96b80","Type":"ContainerDied","Data":"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"}
Sep 08 16:51:35.573278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.572954    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr" event={"ID":"a790194b-f439-4f21-9c85-88ac0fd96b80","Type":"ContainerDied","Data":"2f501d3943907a575dd664659baceac2a80b0fb1e3ff1501ab05d981f30d14e2"}
Sep 08 16:51:35.573278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.572977    2571 scope.go:117] "RemoveContainer" containerID="a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"
Sep 08 16:51:35.582090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.582069    2571 scope.go:117] "RemoveContainer" containerID="a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"
Sep 08 16:51:35.582363 ip-10-0-135-163 kubenswrapper[2571]: E0908 16:51:35.582346    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601\": container with ID starting with a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601 not found: ID does not exist" containerID="a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"
Sep 08 16:51:35.582407 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.582370    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601"} err="failed to get container status \"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601\": rpc error: code = NotFound desc = could not find container \"a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601\": container with ID starting with a6ff41c3df9801d1e4eb5825e0cf39b87587b0c0ab27f3e0203441e16646c601 not found: ID does not exist"
Sep 08 16:51:35.593663 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.593636    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:35.597869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:35.597846    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-vdzrr"]
Sep 08 16:51:36.887761 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:36.887730    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a790194b-f439-4f21-9c85-88ac0fd96b80" path="/var/lib/kubelet/pods/a790194b-f439-4f21-9c85-88ac0fd96b80/volumes"
Sep 08 16:51:40.698924 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:40.698888    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:46.378511 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:46.378434    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:51:52.974278 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:51:52.974244    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xsdgb"]
Sep 08 16:52:01.138571 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:01.138529    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cztqr_cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c/manager/0.log"
Sep 08 16:52:01.740981 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:01.740945    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jfb2p_ddb04eed-c133-41be-bcc1-09386a1b81b3/copy-manifests/0.log"
Sep 08 16:52:01.748625 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:01.748594    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jfb2p_ddb04eed-c133-41be-bcc1-09386a1b81b3/manager/0.log"
Sep 08 16:52:02.319677 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:02.319640    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7bbd6f9566-jszv5_8a6df7df-2857-485a-814a-9b369abda5e7/maas-api/0.log"
Sep 08 16:52:03.911047 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:03.911018    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zdzzj_293cbeeb-1f98-40f1-8776-f613368c1b92/kuadrant-console-plugin/0.log"
Sep 08 16:52:04.148178 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:04.148148    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xsdgb_9ac54591-ffea-4a02-99fa-939739786af2/limitador/0.log"
Sep 08 16:52:04.270210 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:04.270135    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-822pj_5c1a059d-b5fa-4138-bff8-deb81ce7c068/manager/0.log"
Sep 08 16:52:05.075443 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:05.075410    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/1.log"
Sep 08 16:52:05.975636 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:05.975602    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-cd2z5_2f32a388-f617-43d4-9e32-66bbd6e16073/main/0.log"
Sep 08 16:52:06.404543 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:06.404508    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lbpfb_2a44425b-f61d-494e-b74c-7eda55ab5153/main/0.log"
Sep 08 16:52:11.614607 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:11.614570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:52:11.711726 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:11.711684    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log" limit=1024
Sep 08 16:52:14.621956 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:14.621928    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f6xhh_0e0303ba-eecf-48b6-887a-87fcf3f78535/global-pull-secret-syncer/0.log"
Sep 08 16:52:14.755934 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:14.755900    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8cnpg_00ced490-a938-46a8-856b-07376d703c79/konnectivity-agent/0.log"
Sep 08 16:52:14.880282 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:14.880197    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-163.ec2.internal_590fb2a21abc06eff1f2ccb73d30db1a/haproxy/0.log"
Sep 08 16:52:19.872558 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:19.872535    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zdzzj_293cbeeb-1f98-40f1-8776-f613368c1b92/kuadrant-console-plugin/0.log"
Sep 08 16:52:19.990531 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:19.990501    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xsdgb_9ac54591-ffea-4a02-99fa-939739786af2/limitador/0.log"
Sep 08 16:52:20.078575 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:20.078550    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-822pj_5c1a059d-b5fa-4138-bff8-deb81ce7c068/manager/0.log"
Sep 08 16:52:21.783423 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:21.783393    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2xq8w_711a050c-ab0c-4535-93d4-63153e8544dc/kube-state-metrics/0.log"
Sep 08 16:52:21.823666 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:21.823631    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2xq8w_711a050c-ab0c-4535-93d4-63153e8544dc/kube-rbac-proxy-main/0.log"
Sep 08 16:52:21.868166 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:21.868136    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-2xq8w_711a050c-ab0c-4535-93d4-63153e8544dc/kube-rbac-proxy-self/0.log"
Sep 08 16:52:22.013087 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.013061    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86v6p_b8b17c31-b4fe-42a9-9d48-c62b8806e16a/node-exporter/0.log"
Sep 08 16:52:22.039704 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.039644    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86v6p_b8b17c31-b4fe-42a9-9d48-c62b8806e16a/kube-rbac-proxy/0.log"
Sep 08 16:52:22.066258 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.066239    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86v6p_b8b17c31-b4fe-42a9-9d48-c62b8806e16a/init-textfile/0.log"
Sep 08 16:52:22.342335 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.342311    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/prometheus/0.log"
Sep 08 16:52:22.361197 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.361177    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/config-reloader/0.log"
Sep 08 16:52:22.382003 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.381981    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/thanos-sidecar/0.log"
Sep 08 16:52:22.402140 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.402117    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/kube-rbac-proxy-web/0.log"
Sep 08 16:52:22.420494 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.420472    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/kube-rbac-proxy/0.log"
Sep 08 16:52:22.446069 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.446051    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/kube-rbac-proxy-thanos/0.log"
Sep 08 16:52:22.470679 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.470654    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_368af9e1-f4c3-456e-9ada-ccc8f53dad72/init-config-reloader/0.log"
Sep 08 16:52:22.765364 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.765272    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/thanos-query/0.log"
Sep 08 16:52:22.789869 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.789840    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/kube-rbac-proxy-web/0.log"
Sep 08 16:52:22.816714 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.816685    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/kube-rbac-proxy/0.log"
Sep 08 16:52:22.845857 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.845835    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/prom-label-proxy/0.log"
Sep 08 16:52:22.866424 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.866402    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/kube-rbac-proxy-rules/0.log"
Sep 08 16:52:22.885626 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:22.885588    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85fc6fcffb-l6jxt_2499dbb9-fec4-46f6-9a00-b6c7c27a6425/kube-rbac-proxy-metrics/0.log"
Sep 08 16:52:24.439090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.439058    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"]
Sep 08 16:52:24.439484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.439452    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a790194b-f439-4f21-9c85-88ac0fd96b80" containerName="maas-api"
Sep 08 16:52:24.439484 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.439464    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a790194b-f439-4f21-9c85-88ac0fd96b80" containerName="maas-api"
Sep 08 16:52:24.439561 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.439529    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a790194b-f439-4f21-9c85-88ac0fd96b80" containerName="maas-api"
Sep 08 16:52:24.442954 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.442933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.445860 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.445801    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"openshift-service-ca.crt\""
Sep 08 16:52:24.447127 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.447102    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"kube-root-ca.crt\""
Sep 08 16:52:24.447264 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.447102    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tfrvf\"/\"default-dockercfg-924bj\""
Sep 08 16:52:24.453227 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.453204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"]
Sep 08 16:52:24.491058 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.491027    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-lib-modules\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.491169 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.491090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-sys\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.491169 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.491140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-podres\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.491250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.491212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-proc\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.491250 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.491236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6z7\" (UniqueName: \"kubernetes.io/projected/eaa8f77e-0091-4114-9e9a-6049215ab3e7-kube-api-access-ql6z7\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.591893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.591860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-sys\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.591893 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.591898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-podres\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.591929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-proc\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.591958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6z7\" (UniqueName: \"kubernetes.io/projected/eaa8f77e-0091-4114-9e9a-6049215ab3e7-kube-api-access-ql6z7\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.591984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-sys\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.592000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-lib-modules\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.592031    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-podres\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.592066    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-proc\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.592113 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.592105    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eaa8f77e-0091-4114-9e9a-6049215ab3e7-lib-modules\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.601512 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.601492    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6z7\" (UniqueName: \"kubernetes.io/projected/eaa8f77e-0091-4114-9e9a-6049215ab3e7-kube-api-access-ql6z7\") pod \"perf-node-gather-daemonset-gx95f\" (UID: \"eaa8f77e-0091-4114-9e9a-6049215ab3e7\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:24.753742 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:24.753646    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:25.025519 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:25.025496    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"]
Sep 08 16:52:25.027221 ip-10-0-135-163 kubenswrapper[2571]: W0908 16:52:25.027190    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeaa8f77e_0091_4114_9e9a_6049215ab3e7.slice/crio-e1cc92e97e1eb5e40c886c223d51e818a16d94a45bbe2232173a55d576045c5b WatchSource:0}: Error finding container e1cc92e97e1eb5e40c886c223d51e818a16d94a45bbe2232173a55d576045c5b: Status 404 returned error can't find the container with id e1cc92e97e1eb5e40c886c223d51e818a16d94a45bbe2232173a55d576045c5b
Sep 08 16:52:25.808054 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:25.808018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f" event={"ID":"eaa8f77e-0091-4114-9e9a-6049215ab3e7","Type":"ContainerStarted","Data":"2cf4f542b0f09af4a2bb2d9d06546bd7ccebe98b899b837812a622a8b3854644"}
Sep 08 16:52:25.808450 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:25.808069    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f" event={"ID":"eaa8f77e-0091-4114-9e9a-6049215ab3e7","Type":"ContainerStarted","Data":"e1cc92e97e1eb5e40c886c223d51e818a16d94a45bbe2232173a55d576045c5b"}
Sep 08 16:52:25.808450 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:25.808100    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:25.825638 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:25.825591    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f" podStartSLOduration=1.825574548 podStartE2EDuration="1.825574548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:25.824178512 +0000 UTC m=+2375.468255204" watchObservedRunningTime="2026-09-08 16:52:25.825574548 +0000 UTC m=+2375.469651234"
Sep 08 16:52:27.158170 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.158123    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdhsw_e6a6631c-8c75-47b9-aec8-2bafa218cc67/dns/0.log"
Sep 08 16:52:27.175933 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.175891    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cdhsw_e6a6631c-8c75-47b9-aec8-2bafa218cc67/kube-rbac-proxy/0.log"
Sep 08 16:52:27.334986 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.334949    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vw7sb_0e3586bf-382f-44e6-90bd-7dd42f1489f8/dns-node-resolver/0.log"
Sep 08 16:52:27.841946 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.841919    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5789b9fbd6-4kdw6_9065f588-b363-4690-a1ed-899fbc692584/registry/0.log"
Sep 08 16:52:27.857811 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.857785    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5789b9fbd6-4kdw6_9065f588-b363-4690-a1ed-899fbc692584/registry/1.log"
Sep 08 16:52:27.925859 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:27.925833    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ss6b4_b40d2ffe-0c82-4b9c-9cd0-26c8b17a2574/node-ca/0.log"
Sep 08 16:52:29.370090 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:29.370052    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/0.log"
Sep 08 16:52:29.387685 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:29.387651    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/1.log"
Sep 08 16:52:30.407995 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:30.407958    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqgfn_ded0bbca-b5d7-4dce-80ad-44ce042b3c48/serve-healthcheck-canary/0.log"
Sep 08 16:52:30.973351 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:30.973322    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pthx9_9267b4c3-cb2b-4882-8c54-e0d692155618/kube-rbac-proxy/0.log"
Sep 08 16:52:30.994249 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:30.994220    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pthx9_9267b4c3-cb2b-4882-8c54-e0d692155618/exporter/0.log"
Sep 08 16:52:31.015515 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:31.015480    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pthx9_9267b4c3-cb2b-4882-8c54-e0d692155618/extractor/0.log"
Sep 08 16:52:31.823389 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:31.823361    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-gx95f"
Sep 08 16:52:33.350368 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:33.350338    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cztqr_cbcf1f8f-8ed4-465b-b7a5-57b6dee6749c/manager/0.log"
Sep 08 16:52:33.968302 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:33.968204    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jfb2p_ddb04eed-c133-41be-bcc1-09386a1b81b3/manager/0.log"
Sep 08 16:52:33.991749 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:33.991717    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jfb2p_ddb04eed-c133-41be-bcc1-09386a1b81b3/copy-manifests/0.log"
Sep 08 16:52:35.473683 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:35.473656    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78fd8475-d8w8l_446d93c8-7319-4872-9910-ba31fb9c1703/manager/0.log"
Sep 08 16:52:36.485359 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:36.485323    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/0.log"
Sep 08 16:52:36.500541 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:36.500506    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/1.log"
Sep 08 16:52:38.515421 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:38.515389    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/0.log"
Sep 08 16:52:38.532074 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:38.532028    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hhl7g_02036e9c-f04b-4f6d-aa6e-7e45d1bfbf38/istio-proxy/1.log"
Sep 08 16:52:41.917403 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:41.917367    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/kube-multus-additional-cni-plugins/0.log"
Sep 08 16:52:41.940312 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:41.940286    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/egress-router-binary-copy/0.log"
Sep 08 16:52:41.960585 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:41.960561    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/cni-plugins/0.log"
Sep 08 16:52:41.980317 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:41.980298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/bond-cni-plugin/0.log"
Sep 08 16:52:42.000330 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:42.000300    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/routeoverride-cni/0.log"
Sep 08 16:52:42.021762 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:42.021741    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/whereabouts-cni-bincopy/0.log"
Sep 08 16:52:42.041593 ip-10-0-135-163 kubenswrapper[2571]: I0908 16:52:42.041576    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kbt7b_434f72bb-ef9a-4917-8bf1-bf0ff3e9258b/whereabouts-cni/0.log"