Sep 05 19:43:08.307886 ip-10-0-129-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 19:43:08.661993 ip-10-0-129-128 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:08.661993 ip-10-0-129-128 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 19:43:08.661993 ip-10-0-129-128 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:08.661993 ip-10-0-129-128 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 19:43:08.661993 ip-10-0-129-128 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:08.663741 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.663652    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 19:43:08.668070 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668048    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.668070 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668066    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.668070 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668071    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.668070 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668076    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.668070 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668079    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668082    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668086    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668089    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668092    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668095    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668097    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668100    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668103    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668105    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668108    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668111    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668114    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668116    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668119    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668122    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668124    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668127    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668129    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668132    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.668255 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668134    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668137    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668143    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668145    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668148    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668151    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668153    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668156    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668159    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668161    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668164    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668166    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668170    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668175    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668180    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668183    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668186    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668190    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668194    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.668758 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668197    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668200    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668203    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668206    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668208    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668211    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668215    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668218    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668222    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668224    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668227    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668229    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668232    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668234    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668237    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668239    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668242    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668244    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668247    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.669210 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668250    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668253    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668256    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668258    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668262    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668265    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668268    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668270    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668273    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668276    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668279    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668281    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668284    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668286    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668289    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668292    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668294    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668297    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668300    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668302    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.669692 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668305    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668308    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668310    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668313    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668316    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668319    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668321    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668323    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668326    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668329    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668332    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668334    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668337    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668340    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668343    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668345    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.668348    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669053    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669060    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669063    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.670174 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669067    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669069    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669072    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669075    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669077    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669080    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669084    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669086    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669089    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669091    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669094    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669097    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669100    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669102    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669106    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669108    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669111    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669114    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669117    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669120    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.670700 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669122    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669125    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669128    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669132    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669135    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669138    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669140    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669143    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669146    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669148    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669151    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669154    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669156    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669159    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669161    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669164    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669167    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669169    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669172    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669174    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.671192 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669177    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669179    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669182    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669184    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669187    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669189    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669191    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669194    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669197    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669200    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669203    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669205    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669207    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669210    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669213    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669217    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669222    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669227    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669231    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.671691 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669234    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669237    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669240    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669243    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669247    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669249    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669252    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669255    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669257    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669260    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669263    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669265    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669268    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669270    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669273    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669275    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669278    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669280    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669286    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669288    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.672166 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669291    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669296    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669301    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669305    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669308    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669310    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669313    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669315    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669319    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669324    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669326    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669329    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669331    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669334    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669337    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669341    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669344    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669742    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669746    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.672666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669749    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669753    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669755    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669758    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669761    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669765    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669768    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669771    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669775    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669777    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669780    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669783    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669786    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669788    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669792    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669794    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669797    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669800    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669803    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.673144 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669806    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669809    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669812    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669816    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669818    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669821    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669823    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669826    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669828    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669831    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669833    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669836    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669838    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669841    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669843    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669846    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669848    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669851    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669853    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.673710 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669856    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669859    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669862    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669864    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669867    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669869    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669872    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669874    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669877    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669879    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669882    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669884    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669887    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669890    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669892    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669895    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669898    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669901    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669904    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669906    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.674169 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669909    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669912    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669914    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669917    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669919    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669922    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669925    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669927    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669931    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669934    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669936    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669939    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669943    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669946    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669949    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669951    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669954    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669956    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669959    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669961    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.674666 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669964    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669966    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669971    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669974    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669977    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669980    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669983    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669986    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669989    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669991    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669994    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669997    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.669999    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670002    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670004    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670007    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670009    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670012    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670014    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670087    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 19:43:08.675148 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670119    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670125    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670129    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670138    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670141    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670145    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670150    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670153    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670156    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670160    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670163    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670166    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670169    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670172    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670175    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670179    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670182    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670186    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670189    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670192    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670195    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670198    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670202    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 19:43:08.675665 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670204    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670207    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670211    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670213    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670216    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670219    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670222    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670226    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670229    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670232    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670235    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670238    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670246    2575 flags.go:64] FLAG: --enable-server="true"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670248    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670253    2575 flags.go:64] FLAG: --event-burst="100"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670256    2575 flags.go:64] FLAG: --event-qps="50"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670259    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670262    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670265    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670268    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670271    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670274    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670277    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670279    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670284    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 19:43:08.676472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670287    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670291    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670294    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670297    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670299    2575 flags.go:64] FLAG: --feature-gates=""
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670303    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670306    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670309    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670312    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670315    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670318    2575 flags.go:64] FLAG: --help="false"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670321    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670324    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670327    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670330    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670333    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670336    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670339    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670342    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670347    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670350    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670353    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670356    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670359    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 05 19:43:08.677151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670361    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670364    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670367    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670370    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670373    2575 flags.go:64] FLAG: --lock-file=""
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670376    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670378    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670383    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670388    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670392    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670395    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670397    2575 flags.go:64] FLAG: --logging-format="text"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670400    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670404    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670406    2575 flags.go:64] FLAG: --manifest-url=""
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670409    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670413    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670416    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670420    2575 flags.go:64] FLAG: --max-pods="110"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670423    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670426    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670429    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670432    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670435    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670438    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 19:43:08.677768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670441    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670448    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670452    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670455    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670458    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670461    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670466    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670469    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670472    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670476    2575 flags.go:64] FLAG: --port="10250"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670479    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670482    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06fb0206ec1f0997f"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670485    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670488    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670491    2575 flags.go:64] FLAG: --register-node="true"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670495    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670499    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670502    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670505    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670508    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670511    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670514    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670518    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670520    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670523    2575 flags.go:64] FLAG: --runonce="false"
Sep 05 19:43:08.678369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670526    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670529    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670532    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670535    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670538    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670541    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670544    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670547    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670549    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670554    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670556    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670559    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670562    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670565    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670568    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670585    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670588    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670591    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670595    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670598    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670601    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670604    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670608    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670611    2575 flags.go:64] FLAG: --v="2"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670615    2575 flags.go:64] FLAG: --version="false"
Sep 05 19:43:08.679018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670619    2575 flags.go:64] FLAG: --vmodule=""
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670623    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.670627    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670739    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670742    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670746    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670749    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670752    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670754    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670757    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670760    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670764    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670767    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670770    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670773    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670776    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670780    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670782    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670785    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.679640 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670787    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670790    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670792    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670795    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670798    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670801    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670804    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670806    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670809    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670811    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670817    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670819    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670822    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670824    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670833    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670837    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670840    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670842    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670845    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.680113 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670856    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670858    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670861    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670864    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670866    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670869    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670872    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670874    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670877    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670880    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670884    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670886    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670889    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670892    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670894    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670897    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670899    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670902    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670905    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670907    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.680590 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670910    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670913    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670915    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670919    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670921    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670924    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670927    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670930    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670934    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670937    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670939    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670942    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670944    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670947    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670951    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670954    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670957    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670960    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670963    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.681073 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670965    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670968    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670971    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670975    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670979    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670982    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670984    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670987    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670990    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670993    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670995    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.670998    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671001    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671004    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671006    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671009    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671012    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671015    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671018    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671020    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.681541 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671023    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671026    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671029    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671032    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.671034    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.671461    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.678665    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.678679    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.679417    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.679424    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679503    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679507    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679511    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679514    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679517    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.682050 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679521    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679523    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679527    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679529    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679532    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679535    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679538    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679540    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679543    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679546    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679548    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679552    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679556    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679559    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679562    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679564    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679567    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679570    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679587    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679591    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.682465 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679594    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679597    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679599    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679602    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679605    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679608    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679611    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679613    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679616    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679619    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679621    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679624    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679627    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679629    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679632    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679635    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679638    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679640    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679643    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679646    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.682973 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679649    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679652    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679655    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679657    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679660    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679662    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679665    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679669    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679673    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679675    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679678    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679682    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679691    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679694    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679697    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679701    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679705    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679707    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679710    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.683464 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679713    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679716    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679718    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679721    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679724    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679726    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679729    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679731    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679734    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679736    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679739    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679742    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679744    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679747    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679749    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679752    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679754    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679758    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679762    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.683954 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679771    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679775    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679778    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679781    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679784    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679786    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679789    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679798    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679800    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679809    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679812    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679814    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679817    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679819    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679822    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.684409 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679825    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.679830    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679972    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679977    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679980    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679983    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679986    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679989    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679992    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679994    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.679997    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680000    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680003    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680005    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680008    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:08.684799 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680011    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680014    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680016    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680019    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680021    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680024    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680027    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680030    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680032    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680035    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680043    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680046    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680049    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680051    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680054    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680056    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680059    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680062    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680065    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:08.685167 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680067    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680070    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680073    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680075    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680078    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680080    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680083    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680085    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680088    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680090    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680093    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680096    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680098    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680101    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680103    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680106    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680109    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680111    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680114    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680117    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:08.685703 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680121    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680124    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680126    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680129    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680137    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680141    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680143    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680146    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680148    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680151    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680153    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680156    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680159    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680161    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680164    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680166    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680169    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680171    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680174    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680177    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:08.686196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680179    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680182    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680185    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680187    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680190    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680192    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680195    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680197    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680200    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680202    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680205    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680207    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680210    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680215    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680219    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680222    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680225    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680234    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680237    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:08.686689 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680240    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680243    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680245    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680249    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680253    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680256    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680258    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:08.680261    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.680266    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.680357    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.683318    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.684252    2575 server.go:1019] "Starting client certificate rotation"
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.684389    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 19:43:08.687139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.684439    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 19:43:08.687563 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.687546    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 19:43:08.689203 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.689182    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 19:43:08.698145 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.698128    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 05 19:43:08.701063 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.701045    2575 log.go:25] "Validated CRI v1 image API"
Sep 05 19:43:08.701924 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.701909    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 19:43:08.706195 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.706176    2575 fs.go:135] Filesystem UUIDs: map[29ffbf2a-5ac4-4e15-847f-6433e6a38333:/dev/nvme0n1p3 393f8889-df1d-4a40-a1ee-302a81a0f1d4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 05 19:43:08.706244 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.706195    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 19:43:08.710980 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.710873    2575 manager.go:216] Machine: {Timestamp:2026-09-05 19:43:08.710402488 +0000 UTC m=+0.300085752 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094793 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b0411c1e727fa04349fe1134e277 SystemUUID:ec28b041-1c1e-727f-a043-49fe1134e277 BootID:60e1ae66-8132-4829-8842-463e0d569a66 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9e:08:54:64:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9e:08:54:64:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:fd:fa:80:81:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 19:43:08.710980 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.710976    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 19:43:08.711106 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.711049    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 19:43:08.714064 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.714037    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 19:43:08.714197 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.714068    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 19:43:08.714879 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.714865    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 19:43:08.714879 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.714880    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 19:43:08.715010 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.714888    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 19:43:08.715534 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.715524    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 19:43:08.717036 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.717019    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 19:43:08.717073 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.717023    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 19:43:08.721276 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721263    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 19:43:08.721398 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721389    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 19:43:08.721443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721420    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 19:43:08.721443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721429    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 19:43:08.721443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721443    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 19:43:08.721557 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721453    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 05 19:43:08.721557 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721463    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 19:43:08.721939 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721915    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 19:43:08.722015 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.721947    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 19:43:08.724129 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.724115    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 19:43:08.726710 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726692    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 19:43:08.726754 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726734    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 19:43:08.726914 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726904    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 19:43:08.726946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726920    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 19:43:08.726946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726926    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 19:43:08.726946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726931    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 19:43:08.726946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726939    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 19:43:08.726946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.726946    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 19:43:08.727431 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727423    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 19:43:08.727462 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727431    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 19:43:08.727462 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727438    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 19:43:08.727462 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727443    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 19:43:08.727462 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727457    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 19:43:08.727568 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727464    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 19:43:08.727568 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727486    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 19:43:08.727568 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.727491    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 19:43:08.731217 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.731196    2575 server.go:1268] "Started kubelet"
Sep 05 19:43:08.731299 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.731240    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 19:43:08.731423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.731347    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 19:43:08.731514 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.731500    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 19:43:08.732243 ip-10-0-129-128 systemd[1]: Started Kubernetes Kubelet.
Sep 05 19:43:08.733866 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.733846    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 19:43:08.734092 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.734072    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 19:43:08.734696 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.734665    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 19:43:08.734696 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.734666    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 19:43:08.736769 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.736734    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 19:43:08.737049 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.737033    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 19:43:08.739126 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.738234    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-128.ec2.internal.18d2843a7b4d8784  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-128.ec2.internal,UID:ip-10-0-129-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-128.ec2.internal,},FirstTimestamp:2026-09-05 19:43:08.731156356 +0000 UTC m=+0.320839613,LastTimestamp:2026-09-05 19:43:08.731156356 +0000 UTC m=+0.320839613,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-128.ec2.internal,}"
Sep 05 19:43:08.739269 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.739251    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 19:43:08.739849 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.739828    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 19:43:08.740029 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.740011    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:08.740486 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740468    2575 factory.go:55] Registering systemd factory
Sep 05 19:43:08.740486 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740487    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 05 19:43:08.740631 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740559    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 19:43:08.740631 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740559    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 19:43:08.740631 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740597    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 19:43:08.740738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740662    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 19:43:08.740738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740672    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 19:43:08.740738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740695    2575 factory.go:153] Registering CRI-O factory
Sep 05 19:43:08.740738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740705    2575 factory.go:223] Registration of the crio container factory successfully
Sep 05 19:43:08.740838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740750    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 19:43:08.740838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740763    2575 factory.go:103] Registering Raw factory
Sep 05 19:43:08.740838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.740773    2575 manager.go:1196] Started watching for new ooms in manager
Sep 05 19:43:08.741097 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.741085    2575 manager.go:349] Starting recovery of all containers
Sep 05 19:43:08.741200 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.741182    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcmpf"
Sep 05 19:43:08.742257 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.742204    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 19:43:08.742382 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.742364    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 19:43:08.750111 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.749825    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcmpf"
Sep 05 19:43:08.752416 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.752399    2575 manager.go:354] Recovery completed
Sep 05 19:43:08.757330 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.757317    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.761188 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761171    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.761265 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761201    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.761265 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761212    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.761826 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761812    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 19:43:08.761826 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761825    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 19:43:08.761933 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761839    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 19:43:08.761933 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.761845    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.763305 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763292    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.763359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763324    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.763359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763332    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.763904 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763891    2575 policy_none.go:49] "None policy: Start"
Sep 05 19:43:08.763962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763909    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 19:43:08.763962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.763919    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 19:43:08.765896 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.765886    2575 policy_none.go:47] "Start"
Sep 05 19:43:08.796128 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796112    2575 manager.go:337] "Starting Device Plugin manager"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.796145    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796154    2575 server.go:87] "Starting device plugin registration server"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796451    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796466    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796557    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796654    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.796664    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.797228    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 19:43:08.812237 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.797281    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:08.862258 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.862233    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 19:43:08.863549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.863533    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 19:43:08.863635 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.863603    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 19:43:08.863682 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.863652    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 19:43:08.863724 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.863709    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 19:43:08.865814 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.865794    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:08.897534 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.897506    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.898272 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.898253    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.898346 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.898282    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.898346 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.898297    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.898346 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.898322    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.906713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.906700    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.906780 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.906719    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-128.ec2.internal\": node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:08.922481 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.922413    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:08.964484 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.964461    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"]
Sep 05 19:43:08.964557 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.964516    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.966410 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.966395    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.966459 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.966426    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.966459 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.966439    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.968658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.968645    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.968786 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.968775    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.968819 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.968801    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.969356 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969337    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.969437 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969361    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.969437 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969372    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.969437 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969385    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.969437 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969402    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.969568 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.969441    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.971551 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.971538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.971604 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.971562    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:08.972239 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.972225    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:08.972302 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.972249    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:08.972302 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:08.972264    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:08.986093 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.986073    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-128.ec2.internal\" not found" node="ip-10-0-129-128.ec2.internal"
Sep 05 19:43:08.989293 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:08.989278    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-128.ec2.internal\" not found" node="ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.022855 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:09.022837    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:09.042949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.042927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b7f8ba7b88ef6444d6b7d4c2ce3a73-config\") pod \"kube-apiserver-proxy-ip-10-0-129-128.ec2.internal\" (UID: \"75b7f8ba7b88ef6444d6b7d4c2ce3a73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.043026 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.042958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.043026 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.042982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.123069 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:09.123041    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:09.143266 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b7f8ba7b88ef6444d6b7d4c2ce3a73-config\") pod \"kube-apiserver-proxy-ip-10-0-129-128.ec2.internal\" (UID: \"75b7f8ba7b88ef6444d6b7d4c2ce3a73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.143366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.143366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.143366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b7f8ba7b88ef6444d6b7d4c2ce3a73-config\") pod \"kube-apiserver-proxy-ip-10-0-129-128.ec2.internal\" (UID: \"75b7f8ba7b88ef6444d6b7d4c2ce3a73\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.143503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.143503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.143427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bc0f67290e54d294ff1192735899522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal\" (UID: \"1bc0f67290e54d294ff1192735899522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.223678 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:09.223603    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:09.289038 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.289015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.292624 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:09.292604    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:09.324342 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:09.324310    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:09.424898 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:09.424852    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:11.698033 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:11.697948    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:11.798544 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:11.798509    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-128.ec2.internal\" not found"
Sep 05 19:43:11.857017 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.856984    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 19:43:11.857532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.857148    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 19:43:11.891291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.891259    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:11.894699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.894674    2575 apiserver.go:52] "Watching apiserver"
Sep 05 19:43:11.903597 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.903570    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 19:43:11.904004 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.903981    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xt85s","openshift-dns/node-resolver-h7bb2","openshift-image-registry/node-ca-dzr5g","openshift-multus/multus-additional-cni-plugins-6zb8q","openshift-multus/multus-xhprs","openshift-multus/network-metrics-daemon-f82tc","openshift-network-diagnostics/network-check-target-f7gjk","openshift-network-operator/iptables-alerter-55fqp","kube-system/konnectivity-agent-chcpw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f","openshift-cluster-node-tuning-operator/tuned-jl5mm","openshift-ovn-kubernetes/ovnkube-node-nbhxh"]
Sep 05 19:43:11.906568 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.906547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:11.906673 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:11.906618    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:11.908694 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.908677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:11.910885 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.910864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:11.912153 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.912128    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 19:43:11.912357 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.912336    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ms42p\""
Sep 05 19:43:11.912357 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.912346    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.912476 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.912439    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.912605 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.912581    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:11.913243 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.913220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.915443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.915426    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.917560 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.917544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:11.917641 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:11.917596    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:11.917806 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.917791    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.917882 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.917824    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.917954 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.917936    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.917954 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.917949    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 19:43:11.918095 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918081    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 19:43:11.918246 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918231    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 19:43:11.918292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918271    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzh4k\""
Sep 05 19:43:11.918292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918235    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 19:43:11.918356 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918296    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 19:43:11.918422 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918409    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5vbt8\""
Sep 05 19:43:11.918488 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918471    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.918581 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918568    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8l74k\""
Sep 05 19:43:11.918911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918895    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a9507e7-ca6e-41ff-b451-a22c805f69e9-serviceca\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:11.918983 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cnibin\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.918983 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.918983 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-k8s-cni-cncf-io\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.918983 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.918964    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-hostroot\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-daemon-config\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-cni-binary-copy\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-kubelet\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-dbus\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-cnibin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-conf-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-multus-certs\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70e9649-272e-43b4-a83e-730d12d26234-tmp-dir\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgb6\" (UniqueName: \"kubernetes.io/projected/c70e9649-272e-43b4-a83e-730d12d26234-kube-api-access-wjgb6\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919263    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-etc-kubernetes\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g4f\" (UniqueName: \"kubernetes.io/projected/0973d75e-376f-4828-a73a-9282fbbd8a9d-kube-api-access-z9g4f\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v254g\" (UniqueName: \"kubernetes.io/projected/9a9507e7-ca6e-41ff-b451-a22c805f69e9-kube-api-access-v254g\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcb9\" (UniqueName: \"kubernetes.io/projected/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-kube-api-access-ckcb9\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-system-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919419    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-multus\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:11.919522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a9507e7-ca6e-41ff-b451-a22c805f69e9-host\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919569    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-socket-dir-parent\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-netns\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-kubelet-config\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919642    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919665    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-bin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.919703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c70e9649-272e-43b4-a83e-730d12d26234-hosts-file\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:11.919949 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:11.919720    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:11.921847 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.921832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:11.924062 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.924028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:11.924600 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.924583    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.924863 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.924850    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.925086 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.925073    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg6jh\""
Sep 05 19:43:11.925166 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.925149    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 19:43:11.926152 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.926092    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 19:38:10 +0000 UTC" deadline="2028-06-03 17:48:18.768003246 +0000 UTC"
Sep 05 19:43:11.926152 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.926150    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15286h5m6.841855486s"
Sep 05 19:43:11.926291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.926281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:11.927540 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.927524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 19:43:11.928617 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.928602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:11.929020 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929001    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c2ggz\""
Sep 05 19:43:11.929262 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929246    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 19:43:11.929262 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929263    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 19:43:11.929532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929518    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.929688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929676    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.929735 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.929717    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5n9z\""
Sep 05 19:43:11.931375 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.931265    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.931375 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.931267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:11.931656 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.931639    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.933478 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.933460    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal"]
Sep 05 19:43:11.933816 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.933801    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 19:43:11.933918 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.933904    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"
Sep 05 19:43:11.934166 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934151    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 19:43:11.934407 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 19:43:11.934500 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934490    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 19:43:11.934705 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934692    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 19:43:11.934790 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934775    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4q8d\""
Sep 05 19:43:11.934841 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.934814    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 19:43:11.935804 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.935786    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nxtk4\""
Sep 05 19:43:11.935881 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.935825    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 19:43:11.937992 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.937964    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 19:43:11.944587 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.944570    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal"]
Sep 05 19:43:11.945480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.945465    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 19:43:11.955037 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.954984    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8jq7"
Sep 05 19:43:11.963793 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:11.963774    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8jq7"
Sep 05 19:43:12.014448 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.014432    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 19:43:12.020788 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a9507e7-ca6e-41ff-b451-a22c805f69e9-host\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.020875 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.020875 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-socket-dir-parent\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.020875 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a9507e7-ca6e-41ff-b451-a22c805f69e9-host\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-kubelet-config\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-kubelet-config\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbjm\" (UniqueName: \"kubernetes.io/projected/40555d87-452a-46ac-a4ac-fba76c45fd18-kube-api-access-7hbjm\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-run\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-socket-dir-parent\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-var-lib-kubelet\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.020994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-env-overrides\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.021009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-bin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c70e9649-272e-43b4-a83e-730d12d26234-hosts-file\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-systemd\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-bin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-lib-modules\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6735bb9-81da-42b3-9293-9c88493ca967-konnectivity-ca\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c70e9649-272e-43b4-a83e-730d12d26234-hosts-file\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cnibin\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cnibin\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-hostroot\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-var-lib-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eac89f67-689b-4148-84c9-5d0547a19e3d-iptables-alerter-script\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-hostroot\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.021376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-tmp\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-kubelet\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-modprobe-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-sys\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-kubelet\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-config\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021471    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46l7\" (UniqueName: \"kubernetes.io/projected/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kube-api-access-l46l7\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-multus-certs\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgb6\" (UniqueName: \"kubernetes.io/projected/c70e9649-272e-43b4-a83e-730d12d26234-kube-api-access-wjgb6\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-conf\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-slash\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-etc-kubernetes\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-multus-certs\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.022134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2lq\" (UniqueName: \"kubernetes.io/projected/abd3d4ac-ae97-484e-bc85-39432f7bfa62-kube-api-access-9t2lq\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-etc-kubernetes\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-registration-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcb9\" (UniqueName: \"kubernetes.io/projected/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-kube-api-access-ckcb9\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-system-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lfr\" (UniqueName: \"kubernetes.io/projected/eac89f67-689b-4148-84c9-5d0547a19e3d-kube-api-access-67lfr\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-netns\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021973    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-netd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.021997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-netns\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-log-socket\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022070    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-system-cni-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-device-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-netns\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-host\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022171    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovn-node-metrics-cert\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-sys-fs\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a9507e7-ca6e-41ff-b451-a22c805f69e9-serviceca\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-k8s-cni-cncf-io\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-daemon-config\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-run-k8s-cni-cncf-io\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-ovn\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-node-log\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-script-lib\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z67b\" (UniqueName: \"kubernetes.io/projected/74b6f497-0eb1-4359-9475-b9e52a412eb8-kube-api-access-7z67b\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-kubelet\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-cni-binary-copy\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-dbus\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.023668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysconfig\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-cnibin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-conf-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70e9649-272e-43b4-a83e-730d12d26234-tmp-dir\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-cnibin\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-kubernetes\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a9507e7-ca6e-41ff-b451-a22c805f69e9-serviceca\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a134963-6f04-4a07-ad64-8482ce725d59-dbus\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-tuned\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022722    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-conf-dir\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-socket-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g4f\" (UniqueName: \"kubernetes.io/projected/0973d75e-376f-4828-a73a-9282fbbd8a9d-kube-api-access-z9g4f\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-systemd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-etc-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-bin\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.024215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eac89f67-689b-4148-84c9-5d0547a19e3d-host-slash\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v254g\" (UniqueName: \"kubernetes.io/projected/9a9507e7-ca6e-41ff-b451-a22c805f69e9-kube-api-access-v254g\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-multus\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.022983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-systemd-units\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70e9649-272e-43b4-a83e-730d12d26234-tmp-dir\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6735bb9-81da-42b3-9293-9c88493ca967-agent-certs\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-multus-daemon-config\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0973d75e-376f-4828-a73a-9282fbbd8a9d-cni-binary-copy\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.023285    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0973d75e-376f-4828-a73a-9282fbbd8a9d-host-var-lib-cni-multus\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.023298    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:12.024689 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.023393    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:12.523361475 +0000 UTC m=+1.940434898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:12.028347 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.028330    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 19:43:12.031785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.031766    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgb6\" (UniqueName: \"kubernetes.io/projected/c70e9649-272e-43b4-a83e-730d12d26234-kube-api-access-wjgb6\") pod \"node-resolver-h7bb2\" (UID: \"c70e9649-272e-43b4-a83e-730d12d26234\") " pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.031879 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.031782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcb9\" (UniqueName: \"kubernetes.io/projected/f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f-kube-api-access-ckcb9\") pod \"multus-additional-cni-plugins-6zb8q\" (UID: \"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f\") " pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.033358 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.033334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g4f\" (UniqueName: \"kubernetes.io/projected/0973d75e-376f-4828-a73a-9282fbbd8a9d-kube-api-access-z9g4f\") pod \"multus-xhprs\" (UID: \"0973d75e-376f-4828-a73a-9282fbbd8a9d\") " pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.034260 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.034240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v254g\" (UniqueName: \"kubernetes.io/projected/9a9507e7-ca6e-41ff-b451-a22c805f69e9-kube-api-access-v254g\") pod \"node-ca-dzr5g\" (UID: \"9a9507e7-ca6e-41ff-b451-a22c805f69e9\") " pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.057265 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.057226    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b7f8ba7b88ef6444d6b7d4c2ce3a73.slice/crio-0b5d931def0338b24ad923e4888c85bcb57c910a38eda9d3f925ddae13c5ddbf WatchSource:0}: Error finding container 0b5d931def0338b24ad923e4888c85bcb57c910a38eda9d3f925ddae13c5ddbf: Status 404 returned error can't find the container with id 0b5d931def0338b24ad923e4888c85bcb57c910a38eda9d3f925ddae13c5ddbf
Sep 05 19:43:12.057485 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.057469    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc0f67290e54d294ff1192735899522.slice/crio-21ed493392bd8e60dde8843169e0b9ed7bc7f9ccc9d905a5125e41ddfa973b64 WatchSource:0}: Error finding container 21ed493392bd8e60dde8843169e0b9ed7bc7f9ccc9d905a5125e41ddfa973b64: Status 404 returned error can't find the container with id 21ed493392bd8e60dde8843169e0b9ed7bc7f9ccc9d905a5125e41ddfa973b64
Sep 05 19:43:12.062686 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.062666    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:43:12.123623 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-kubernetes\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.123623 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-tuned\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-socket-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-systemd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-etc-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123721    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-bin\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eac89f67-689b-4148-84c9-5d0547a19e3d-host-slash\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-kubernetes\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-systemd-units\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-systemd-units\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-systemd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123850    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-socket-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123838    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-bin\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eac89f67-689b-4148-84c9-5d0547a19e3d-host-slash\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.123869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123871    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-etc-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6735bb9-81da-42b3-9293-9c88493ca967-agent-certs\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.123994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbjm\" (UniqueName: \"kubernetes.io/projected/40555d87-452a-46ac-a4ac-fba76c45fd18-kube-api-access-7hbjm\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-run\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-var-lib-kubelet\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-env-overrides\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-systemd\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-run\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-lib-modules\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-var-lib-kubelet\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6735bb9-81da-42b3-9293-9c88493ca967-konnectivity-ca\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-var-lib-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eac89f67-689b-4148-84c9-5d0547a19e3d-iptables-alerter-script\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-lib-modules\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-tmp\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.124699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-var-lib-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-modprobe-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-sys\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-config\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46l7\" (UniqueName: \"kubernetes.io/projected/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kube-api-access-l46l7\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-conf\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-slash\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2lq\" (UniqueName: \"kubernetes.io/projected/abd3d4ac-ae97-484e-bc85-39432f7bfa62-kube-api-access-9t2lq\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-registration-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lfr\" (UniqueName: \"kubernetes.io/projected/eac89f67-689b-4148-84c9-5d0547a19e3d-kube-api-access-67lfr\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-netns\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124608    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-env-overrides\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-netd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:12.125475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-log-socket\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-device-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124757    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6735bb9-81da-42b3-9293-9c88493ca967-konnectivity-ca\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124816    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-registration-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-host\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-host\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124853    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eac89f67-689b-4148-84c9-5d0547a19e3d-iptables-alerter-script\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovn-node-metrics-cert\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-cni-netd\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-modprobe-d\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-netns\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-sys-fs\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.124998    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-openvswitch\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-ovn\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.126359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-node-log\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-script-lib\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z67b\" (UniqueName: \"kubernetes.io/projected/74b6f497-0eb1-4359-9475-b9e52a412eb8-kube-api-access-7z67b\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125145    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysctl-conf\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-kubelet\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-sys\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysconfig\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-systemd\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125292    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-sysconfig\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125353    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-log-socket\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125356    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-slash\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-device-dir\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-run-ovn\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-sys-fs\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-host-kubelet\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abd3d4ac-ae97-484e-bc85-39432f7bfa62-node-log\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127056 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.125510    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.125588    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:12.625568936 +0000 UTC m=+2.042642350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-config\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.125828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovnkube-script-lib\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.126686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-etc-tuned\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.127053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6f497-0eb1-4359-9475-b9e52a412eb8-tmp\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.127165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6735bb9-81da-42b3-9293-9c88493ca967-agent-certs\") pod \"konnectivity-agent-chcpw\" (UID: \"d6735bb9-81da-42b3-9293-9c88493ca967\") " pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.127553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.127374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abd3d4ac-ae97-484e-bc85-39432f7bfa62-ovn-node-metrics-cert\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.132586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.132557    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbjm\" (UniqueName: \"kubernetes.io/projected/40555d87-452a-46ac-a4ac-fba76c45fd18-kube-api-access-7hbjm\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.135466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.135440    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2lq\" (UniqueName: \"kubernetes.io/projected/abd3d4ac-ae97-484e-bc85-39432f7bfa62-kube-api-access-9t2lq\") pod \"ovnkube-node-nbhxh\" (UID: \"abd3d4ac-ae97-484e-bc85-39432f7bfa62\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.137166 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.137145    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:12.137166 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.137166    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:12.137346 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.137176    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:12.137346 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.137249    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:12.637237093 +0000 UTC m=+2.054310495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:12.138768 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.138747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z67b\" (UniqueName: \"kubernetes.io/projected/74b6f497-0eb1-4359-9475-b9e52a412eb8-kube-api-access-7z67b\") pod \"tuned-jl5mm\" (UID: \"74b6f497-0eb1-4359-9475-b9e52a412eb8\") " pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.138870 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.138854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lfr\" (UniqueName: \"kubernetes.io/projected/eac89f67-689b-4148-84c9-5d0547a19e3d-kube-api-access-67lfr\") pod \"iptables-alerter-55fqp\" (UID: \"eac89f67-689b-4148-84c9-5d0547a19e3d\") " pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.139558 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.139543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46l7\" (UniqueName: \"kubernetes.io/projected/6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56-kube-api-access-l46l7\") pod \"aws-ebs-csi-driver-node-xbs7f\" (UID: \"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.190628 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.190598    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:12.234320 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.234238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h7bb2"
Sep 05 19:43:12.240546 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.240526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzr5g"
Sep 05 19:43:12.244825 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.244800    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e9649_272e_43b4_a83e_730d12d26234.slice/crio-b03d23cffadd69d0c81216bedd4ba2d60206d5b818d79315f3399af1193841a6 WatchSource:0}: Error finding container b03d23cffadd69d0c81216bedd4ba2d60206d5b818d79315f3399af1193841a6: Status 404 returned error can't find the container with id b03d23cffadd69d0c81216bedd4ba2d60206d5b818d79315f3399af1193841a6
Sep 05 19:43:12.248040 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.248013    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zb8q"
Sep 05 19:43:12.249782 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.249752    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9507e7_ca6e_41ff_b451_a22c805f69e9.slice/crio-b6ec0544852532167e9b79c750e6b57d9c4dc314a1f459f7b474259cf0c9c83a WatchSource:0}: Error finding container b6ec0544852532167e9b79c750e6b57d9c4dc314a1f459f7b474259cf0c9c83a: Status 404 returned error can't find the container with id b6ec0544852532167e9b79c750e6b57d9c4dc314a1f459f7b474259cf0c9c83a
Sep 05 19:43:12.258391 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.258230    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58c4c6b_5ea4_42b0_b6bf_e57d5b97582f.slice/crio-a23cc7f97535b77971737876c674e119fb93aa612f4788894c00ff53fe1f9f3b WatchSource:0}: Error finding container a23cc7f97535b77971737876c674e119fb93aa612f4788894c00ff53fe1f9f3b: Status 404 returned error can't find the container with id a23cc7f97535b77971737876c674e119fb93aa612f4788894c00ff53fe1f9f3b
Sep 05 19:43:12.272509 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.272487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhprs"
Sep 05 19:43:12.280215 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.280168    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0973d75e_376f_4828_a73a_9282fbbd8a9d.slice/crio-eedb68ca0eccda16041a5de9faeafd9a5221845ab70d7b5bfaf5540cc77d8ad3 WatchSource:0}: Error finding container eedb68ca0eccda16041a5de9faeafd9a5221845ab70d7b5bfaf5540cc77d8ad3: Status 404 returned error can't find the container with id eedb68ca0eccda16041a5de9faeafd9a5221845ab70d7b5bfaf5540cc77d8ad3
Sep 05 19:43:12.286154 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.286136    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55fqp"
Sep 05 19:43:12.293853 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.293828    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac89f67_689b_4148_84c9_5d0547a19e3d.slice/crio-612a0557e64610e45e1973ac982d4b7381e240e54b0228ff2e6bb56a2006cb1a WatchSource:0}: Error finding container 612a0557e64610e45e1973ac982d4b7381e240e54b0228ff2e6bb56a2006cb1a: Status 404 returned error can't find the container with id 612a0557e64610e45e1973ac982d4b7381e240e54b0228ff2e6bb56a2006cb1a
Sep 05 19:43:12.307499 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.307480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:12.315564 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.315546    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6735bb9_81da_42b3_9293_9c88493ca967.slice/crio-bfd2668cffcd6b70c3830b70b360dacf8ab0cc23d594d501338361cd9e0ae31f WatchSource:0}: Error finding container bfd2668cffcd6b70c3830b70b360dacf8ab0cc23d594d501338361cd9e0ae31f: Status 404 returned error can't find the container with id bfd2668cffcd6b70c3830b70b360dacf8ab0cc23d594d501338361cd9e0ae31f
Sep 05 19:43:12.317997 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.317980    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:12.319004 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.318987    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f"
Sep 05 19:43:12.326923 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.326898    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee559bd_1ee0_48f9_bd55_3bb6ddbccf56.slice/crio-40aabfc0991eae978170f2e541f7464f0a10146cb2fe510b1a78c4a747d82c30 WatchSource:0}: Error finding container 40aabfc0991eae978170f2e541f7464f0a10146cb2fe510b1a78c4a747d82c30: Status 404 returned error can't find the container with id 40aabfc0991eae978170f2e541f7464f0a10146cb2fe510b1a78c4a747d82c30
Sep 05 19:43:12.331246 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.331230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jl5mm"
Sep 05 19:43:12.339814 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.339791    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b6f497_0eb1_4359_9475_b9e52a412eb8.slice/crio-0a6decf8e4bb6884cd58c8c0a16a14d035d83e4f4020caac135ec2a2876a2f50 WatchSource:0}: Error finding container 0a6decf8e4bb6884cd58c8c0a16a14d035d83e4f4020caac135ec2a2876a2f50: Status 404 returned error can't find the container with id 0a6decf8e4bb6884cd58c8c0a16a14d035d83e4f4020caac135ec2a2876a2f50
Sep 05 19:43:12.342816 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.342796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:12.350688 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:12.350645    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd3d4ac_ae97_484e_bc85_39432f7bfa62.slice/crio-4be9b39cde1899423504ae021bf4c6941c897c0a970776f9e28bb68dec68dc8c WatchSource:0}: Error finding container 4be9b39cde1899423504ae021bf4c6941c897c0a970776f9e28bb68dec68dc8c: Status 404 returned error can't find the container with id 4be9b39cde1899423504ae021bf4c6941c897c0a970776f9e28bb68dec68dc8c
Sep 05 19:43:12.527822 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.527741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:12.527973 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.527901    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:12.528056 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.528016    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:13.52795963 +0000 UTC m=+2.945033045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:12.629047 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.628879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:12.629047 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.629033    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:12.629285 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.629101    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:13.629082328 +0000 UTC m=+3.046155746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:12.730205 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.730152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:12.730383 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.730355    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:12.730383 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.730380    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:12.730489 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.730392    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:12.730489 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:12.730453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:13.730434877 +0000 UTC m=+3.147508278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:12.967106 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.966910    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:11 +0000 UTC" deadline="2028-02-14 03:13:47.472021037 +0000 UTC"
Sep 05 19:43:12.967106 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:12.966949    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12631h30m34.505076744s"
Sep 05 19:43:13.041578 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.040766    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:13.041578 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.040917    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:13.054501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.054454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f" event={"ID":"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56","Type":"ContainerStarted","Data":"40aabfc0991eae978170f2e541f7464f0a10146cb2fe510b1a78c4a747d82c30"}
Sep 05 19:43:13.070193 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.070144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55fqp" event={"ID":"eac89f67-689b-4148-84c9-5d0547a19e3d","Type":"ContainerStarted","Data":"612a0557e64610e45e1973ac982d4b7381e240e54b0228ff2e6bb56a2006cb1a"}
Sep 05 19:43:13.086776 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.086741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhprs" event={"ID":"0973d75e-376f-4828-a73a-9282fbbd8a9d","Type":"ContainerStarted","Data":"eedb68ca0eccda16041a5de9faeafd9a5221845ab70d7b5bfaf5540cc77d8ad3"}
Sep 05 19:43:13.099543 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.099509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerStarted","Data":"a23cc7f97535b77971737876c674e119fb93aa612f4788894c00ff53fe1f9f3b"}
Sep 05 19:43:13.109826 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.109772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h7bb2" event={"ID":"c70e9649-272e-43b4-a83e-730d12d26234","Type":"ContainerStarted","Data":"b03d23cffadd69d0c81216bedd4ba2d60206d5b818d79315f3399af1193841a6"}
Sep 05 19:43:13.114419 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.114390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal" event={"ID":"75b7f8ba7b88ef6444d6b7d4c2ce3a73","Type":"ContainerStarted","Data":"0b5d931def0338b24ad923e4888c85bcb57c910a38eda9d3f925ddae13c5ddbf"}
Sep 05 19:43:13.120923 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.120893    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal" event={"ID":"1bc0f67290e54d294ff1192735899522","Type":"ContainerStarted","Data":"21ed493392bd8e60dde8843169e0b9ed7bc7f9ccc9d905a5125e41ddfa973b64"}
Sep 05 19:43:13.129571 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.129529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"4be9b39cde1899423504ae021bf4c6941c897c0a970776f9e28bb68dec68dc8c"}
Sep 05 19:43:13.132200 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.132166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jl5mm" event={"ID":"74b6f497-0eb1-4359-9475-b9e52a412eb8","Type":"ContainerStarted","Data":"0a6decf8e4bb6884cd58c8c0a16a14d035d83e4f4020caac135ec2a2876a2f50"}
Sep 05 19:43:13.136342 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.136316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chcpw" event={"ID":"d6735bb9-81da-42b3-9293-9c88493ca967","Type":"ContainerStarted","Data":"bfd2668cffcd6b70c3830b70b360dacf8ab0cc23d594d501338361cd9e0ae31f"}
Sep 05 19:43:13.149978 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.149953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzr5g" event={"ID":"9a9507e7-ca6e-41ff-b451-a22c805f69e9","Type":"ContainerStarted","Data":"b6ec0544852532167e9b79c750e6b57d9c4dc314a1f459f7b474259cf0c9c83a"}
Sep 05 19:43:13.447782 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.447574    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:13.536511 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.536471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:13.536700 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.536621    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:13.536700 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.536693    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:15.536675176 +0000 UTC m=+4.953748580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:13.637474 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.637439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:13.637658 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.637594    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:13.637658 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.637655    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:15.637636826 +0000 UTC m=+5.054710244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:13.738201 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.738108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:13.738414 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.738331    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:13.738414 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.738351    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:13.738414 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.738364    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:13.738414 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:13.738415    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:15.738398991 +0000 UTC m=+5.155472397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:13.967430 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.967325    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:11 +0000 UTC" deadline="2028-03-04 21:37:26.041988888 +0000 UTC"
Sep 05 19:43:13.967430 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:13.967390    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13105h54m12.074602988s"
Sep 05 19:43:14.036979 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:14.036897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:14.037131 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:14.037034    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:14.037131 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:14.037115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:14.037278 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:14.037212    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:15.039434 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:15.039389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:15.039871 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.039543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:15.556060 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:15.556018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:15.556323 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.556272    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:15.556408 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.556339    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:19.556319735 +0000 UTC m=+8.973393142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:15.657213 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:15.656756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:15.657213 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.656937    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:15.657213 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.657002    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:19.656983641 +0000 UTC m=+9.074057049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:15.757151 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:15.757108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:15.757372 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.757353    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:15.757435 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.757385    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:15.757435 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.757397    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:15.757533 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:15.757461    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:19.757440525 +0000 UTC m=+9.174513950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:16.037377 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:16.037346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:16.037541 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:16.037484    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:16.037598 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:16.037551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:16.037693 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:16.037672    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:17.036706 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:17.036676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:17.037323 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:17.036805    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:18.037513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:18.037480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:18.037992 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:18.037634    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:18.037992 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:18.037649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:18.037992 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:18.037786    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:19.038804 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:19.037371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:19.038804 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.037496    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:19.589866 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:19.589826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:19.590065 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.589986    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:19.590065 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.590049    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:27.590030415 +0000 UTC m=+17.007103823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:19.691075 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:19.690977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:19.691256 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.691110    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:19.691256 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.691203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:27.69116236 +0000 UTC m=+17.108235781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:19.791508 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:19.791469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:19.791695 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.791670    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:19.791695 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.791694    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:19.791800 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.791706    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:19.791800 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:19.791766    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:27.791747539 +0000 UTC m=+17.208820986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:20.036948 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:20.036839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:20.037117 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:20.036988    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:20.037390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:20.037234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:20.037390 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:20.037349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:21.037634 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:21.037458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:21.037634 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:21.037594    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:22.036814 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:22.036779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:22.036994 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:22.036825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:22.036994 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:22.036894    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:22.037118 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:22.037022    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:23.040021 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:23.037614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:23.040021 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:23.037780    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:24.037249 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:24.037211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:24.037418 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:24.037346    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:24.037418 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:24.037216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:24.037520 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:24.037445    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:25.036627 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:25.036596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:25.037043 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:25.036695    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:26.036800 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:26.036756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:26.037271 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:26.036778    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:26.037271 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:26.036918    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:26.037271 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:26.036944    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:27.037506 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:27.037471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:27.037934 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.037603    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:27.649992 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:27.649947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:27.650253 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.650099    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:27.650253 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.650174    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:43.650152859 +0000 UTC m=+33.067226283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:27.750968 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:27.750931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:27.751148 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.751067    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:27.751148 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.751141    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:43.751120401 +0000 UTC m=+33.168193825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:27.852007 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:27.851933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:27.852217 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.852080    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:27.852217 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.852113    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:27.852217 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.852124    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:27.852217 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:27.852207    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:43.852174566 +0000 UTC m=+33.269247992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:28.036740 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:28.036651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:28.036740 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:28.036707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:28.036951 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:28.036781    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:28.036951 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:28.036899    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:29.037069 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:29.037029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:29.037571 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:29.037199    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:30.037394 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:30.037361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:30.037829 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:30.037369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:30.037829 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:30.037474    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:30.037829 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:30.037595    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:31.037742 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:31.037717    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:31.038084 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:31.037798    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:32.036723 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.036598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:32.036837 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:32.036801    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:32.037117 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.037099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:32.037222 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:32.037168    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:32.197076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.196859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f" event={"ID":"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56","Type":"ContainerStarted","Data":"62572d0a32258d3e8292cc566634b6e9f1ba2dbe877ce3cfeb7635ec9bf241c8"}
Sep 05 19:43:32.200906 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.200877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhprs" event={"ID":"0973d75e-376f-4828-a73a-9282fbbd8a9d","Type":"ContainerStarted","Data":"e4090c1553217b03efb2815dd86724486f6523867e4023a5d224b08c5f7b53bb"}
Sep 05 19:43:32.206537 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.206500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal" event={"ID":"75b7f8ba7b88ef6444d6b7d4c2ce3a73","Type":"ContainerStarted","Data":"09a3894e6011077fcfd00e9ae32fe0a202430020e04ddf0f1cb97dd0de9ad800"}
Sep 05 19:43:32.208890 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.208865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"ac21f06117512f5dcddb544dd1916f502c91a8a8ea4da78744fad20ccf735d51"}
Sep 05 19:43:32.208976 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.208957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"1a81b247f0ebf6df7904627ad1f3eeb6c95ecd576fd7960099487dc8452e371d"}
Sep 05 19:43:32.209031 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.209008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"9463e32e1eec543b53c33b243035d9e6d5f3bd2423ec38983b29db6786fe5fc7"}
Sep 05 19:43:32.210786 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.210739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jl5mm" event={"ID":"74b6f497-0eb1-4359-9475-b9e52a412eb8","Type":"ContainerStarted","Data":"e33746c2e982331a0024f658ec9014ceae082dab797ab81868fa6ebfc8d2bb28"}
Sep 05 19:43:32.212303 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.212239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chcpw" event={"ID":"d6735bb9-81da-42b3-9293-9c88493ca967","Type":"ContainerStarted","Data":"a9cd9f9fe7cf6a1f51ee847b3e031d8c1a1e32fb14ff3a7d139849c65766bc63"}
Sep 05 19:43:32.260960 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.260910    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-128.ec2.internal" podStartSLOduration=21.260895526 podStartE2EDuration="21.260895526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:32.260315388 +0000 UTC m=+21.677388805" watchObservedRunningTime="2026-09-05 19:43:32.260895526 +0000 UTC m=+21.677968949"
Sep 05 19:43:32.261075 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.261041    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dzr5g" podStartSLOduration=6.3001500440000004 podStartE2EDuration="21.261034038s" totalImagesPullingTime="14.960883994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:32.236886883 +0000 UTC m=+21.653960305" watchObservedRunningTime="2026-09-05 19:43:32.261034038 +0000 UTC m=+21.678107465"
Sep 05 19:43:32.285170 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.285120    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xhprs" podStartSLOduration=2.17057691 podStartE2EDuration="21.285099444s" totalImagesPullingTime="19.114522534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:32.280856964 +0000 UTC m=+21.697930388" watchObservedRunningTime="2026-09-05 19:43:32.285099444 +0000 UTC m=+21.702172869"
Sep 05 19:43:32.327954 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.327911    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-chcpw" podStartSLOduration=2.302018895 podStartE2EDuration="21.327892601s" totalImagesPullingTime="19.025873706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:32.302929467 +0000 UTC m=+21.720002891" watchObservedRunningTime="2026-09-05 19:43:32.327892601 +0000 UTC m=+21.744966046"
Sep 05 19:43:32.328067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:32.328043    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jl5mm" podStartSLOduration=2.2901425619999998 podStartE2EDuration="21.328039029s" totalImagesPullingTime="19.037896467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:32.32769027 +0000 UTC m=+21.744763690" watchObservedRunningTime="2026-09-05 19:43:32.328039029 +0000 UTC m=+21.745112451"
Sep 05 19:43:33.037574 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.037486    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:33.037721 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:33.037615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:33.217346 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.217310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55fqp" event={"ID":"eac89f67-689b-4148-84c9-5d0547a19e3d","Type":"ContainerStarted","Data":"65ea8ade0339012c633598f48bc77936ca888fa61a4c0c72c0a13cbf00fa70fb"}
Sep 05 19:43:33.218774 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.218747    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="d69009deb2a3c54a2a69db2e3ca984f4a6a2b78829fa8879bec9295059631fff" exitCode=0
Sep 05 19:43:33.218876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.218834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"d69009deb2a3c54a2a69db2e3ca984f4a6a2b78829fa8879bec9295059631fff"}
Sep 05 19:43:33.220341 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.220324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h7bb2" event={"ID":"c70e9649-272e-43b4-a83e-730d12d26234","Type":"ContainerStarted","Data":"0df75e3f77e2307ac0f1da7ba30d742eaee2a4b4f997bbcf805dc2dd343dbaaf"}
Sep 05 19:43:33.221838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.221820    2575 generic.go:356] "Generic (PLEG): container finished" podID="1bc0f67290e54d294ff1192735899522" containerID="7b24d6ba9a6da41eeec524e6897ef8eb411fb7eedd101f311b03513e5fb21357" exitCode=0
Sep 05 19:43:33.221924 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.221849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal" event={"ID":"1bc0f67290e54d294ff1192735899522","Type":"ContainerDied","Data":"7b24d6ba9a6da41eeec524e6897ef8eb411fb7eedd101f311b03513e5fb21357"}
Sep 05 19:43:33.225049 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.225030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"bd82c95a3d7a6cccd4dc9335356b986aebe05c02fbfad451b77ba14178bc1e10"}
Sep 05 19:43:33.225128 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.225093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"dea10f937f2689d373caf996d138c71e69faab0d9b0c0bdde84246883c1602e7"}
Sep 05 19:43:33.225179 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.225146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"b9b52560576fe08f96abaaf79c81f7d0f5a5587a8af42e17d9bbf8ccf6009efb"}
Sep 05 19:43:33.226434 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.226413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzr5g" event={"ID":"9a9507e7-ca6e-41ff-b451-a22c805f69e9","Type":"ContainerStarted","Data":"d0ee9b6247548a1543f303b8972a3d975c80212526ed77aff7582dd7be2ac924"}
Sep 05 19:43:33.246815 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.246773    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-55fqp" podStartSLOduration=3.251532604 podStartE2EDuration="22.246760463s" totalImagesPullingTime="18.995227859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:33.246631856 +0000 UTC m=+22.663705278" watchObservedRunningTime="2026-09-05 19:43:33.246760463 +0000 UTC m=+22.663833930"
Sep 05 19:43:33.303612 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.303528    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h7bb2" podStartSLOduration=3.160037814 podStartE2EDuration="22.303511543s" totalImagesPullingTime="19.143473729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:33.303266378 +0000 UTC m=+22.720339801" watchObservedRunningTime="2026-09-05 19:43:33.303511543 +0000 UTC m=+22.720584968"
Sep 05 19:43:33.423255 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.423222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:33.520485 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.520454    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 19:43:33.539496 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.539460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:33.984216 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.983955    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T19:43:33.520479123Z","UUID":"6aee4644-270a-406d-a03d-58b23bc1f6ef","Handler":null,"Name":"","Endpoint":""}
Sep 05 19:43:33.985573 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.985551    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 19:43:33.985709 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:33.985588    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 19:43:34.037124 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:34.037095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:34.037124 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:34.037128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:34.037388 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:34.037232    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:34.037388 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:34.037289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:34.231690 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:34.231648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f" event={"ID":"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56","Type":"ContainerStarted","Data":"8fcd103e34b7ac07fc9e62a71283fe55783ecdd4eab32f0bad39bc287093a653"}
Sep 05 19:43:34.233940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:34.233902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal" event={"ID":"1bc0f67290e54d294ff1192735899522","Type":"ContainerStarted","Data":"d8115f3161c0bd8e9224e3b965bab966cc9fecad182638e0fa2b4b069dc68754"}
Sep 05 19:43:35.037328 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.037250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:35.037521 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:35.037370    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:35.239092 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.239059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"b3096dca98707fc8a31054bb1f8632b01b5115804192d25f3ba381ce241436ff"}
Sep 05 19:43:35.241007 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.240978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f" event={"ID":"6ee559bd-1ee0-48f9-bd55-3bb6ddbccf56","Type":"ContainerStarted","Data":"90b7441b63f0b14405f389d1408b971ec7933c75f21abd46661a54b778303f9e"}
Sep 05 19:43:35.261951 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.261898    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbs7f" podStartSLOduration=2.860438717 podStartE2EDuration="24.261884407s" totalImagesPullingTime="21.40144569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:35.261391465 +0000 UTC m=+24.678464901" watchObservedRunningTime="2026-09-05 19:43:35.261884407 +0000 UTC m=+24.678957831"
Sep 05 19:43:35.262360 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.262318    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-128.ec2.internal" podStartSLOduration=24.262281913 podStartE2EDuration="24.262281913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:34.252465542 +0000 UTC m=+23.669538965" watchObservedRunningTime="2026-09-05 19:43:35.262281913 +0000 UTC m=+24.679355352"
Sep 05 19:43:35.518766 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.518734    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:35.519332 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:35.519310    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:36.036464 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:36.036431    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:36.036652 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:36.036559    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:36.036652 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:36.036601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:36.036765 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:36.036703    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:36.246456 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:36.246147    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:36.246456 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:36.246307    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chcpw"
Sep 05 19:43:36.316092 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:36.316061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:37.037514 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:37.037479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:37.037704 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:37.037614    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:38.036789 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.036562    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:38.037141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.036578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:38.037141 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:38.036831    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:38.037141 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:38.036887    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:38.250291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.250250    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="af7fe77a19dddb3c6cbcec96be061373d692342dbf414810cf6d123ff0de247a" exitCode=0
Sep 05 19:43:38.250291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.250290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"af7fe77a19dddb3c6cbcec96be061373d692342dbf414810cf6d123ff0de247a"}
Sep 05 19:43:38.254196 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.254082    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" event={"ID":"abd3d4ac-ae97-484e-bc85-39432f7bfa62","Type":"ContainerStarted","Data":"d62528657c9ce4c1a2ee0556d7c5aaad41e6517e39c376797d15ec4b4d548f91"}
Sep 05 19:43:38.254468 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.254449    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:38.254538 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.254516    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:38.254588 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.254566    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:38.269318 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.269297    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:38.269405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.269390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:43:38.306356 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:38.306281    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh" podStartSLOduration=7.757214198 podStartE2EDuration="27.30626875s" totalImagesPullingTime="19.549054552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:38.304276925 +0000 UTC m=+27.721350347" watchObservedRunningTime="2026-09-05 19:43:38.30626875 +0000 UTC m=+27.723342173"
Sep 05 19:43:39.021320 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.021285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:39.036837 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.036761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:39.037557 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:39.036869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:39.121892 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.121861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:39.258935 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.258841    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="fd027a45ffedf34e04d6060a33be9dbf0686a877001a676a1e9f275c370e32f4" exitCode=0
Sep 05 19:43:39.259100 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.258926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"fd027a45ffedf34e04d6060a33be9dbf0686a877001a676a1e9f275c370e32f4"}
Sep 05 19:43:39.356579 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.356548    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xt85s"]
Sep 05 19:43:39.356743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.356702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:39.356832 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:39.356805    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:39.357798 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.357774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f82tc"]
Sep 05 19:43:39.357894 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.357872    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:39.357981 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:39.357964    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:39.359865 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.359845    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7gjk"]
Sep 05 19:43:39.359948 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.359914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:39.359996 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:39.359973    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:39.508584 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.508546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:39.572798 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:39.572759    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:40.264299 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:40.264036    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="3dc627db89403c51db4b77e2c9763069456d64f53064f1310b1c71ee03cdce10" exitCode=0
Sep 05 19:43:40.264299 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:40.264117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"3dc627db89403c51db4b77e2c9763069456d64f53064f1310b1c71ee03cdce10"}
Sep 05 19:43:40.448399 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:40.448358    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:40.548843 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:40.548743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:40.617412 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:40.617377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:41.038025 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:41.037991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:41.038199 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:41.038098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:41.038199 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:41.038137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:41.038199 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:41.038153    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:41.038370 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:41.038219    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:41.038370 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:41.038277    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:42.815457 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:42.815423    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:42.877798 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:42.877764    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:42.939704 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:42.939668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:43.037414 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.037378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:43.037414 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.037407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:43.037629 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.037501    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:43.037629 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.037553    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:43.037629 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.037599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:43.037795 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.037685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:43.671199 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.671160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:43.671353 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.671331    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:43.671429 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.671414    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret podName:3a134963-6f04-4a07-ad64-8482ce725d59 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:15.671392776 +0000 UTC m=+65.088466276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret") pod "global-pull-secret-syncer-xt85s" (UID: "3a134963-6f04-4a07-ad64-8482ce725d59") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:43.771572 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.771535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:43.771713 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.771683    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:43.771788 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.771767    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:15.771745479 +0000 UTC m=+65.188818897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:43.840750 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.840647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:43.871942 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.871899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:43.872121 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.872078    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:43.872121 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.872108    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:43.872235 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.872124    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vz8m8 for pod openshift-network-diagnostics/network-check-target-f7gjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:43.872235 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:43.872203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8 podName:9b002bc5-39bd-423b-a702-d21ad6573447 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:15.872172021 +0000 UTC m=+65.289245425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vz8m8" (UniqueName: "kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8") pod "network-check-target-f7gjk" (UID: "9b002bc5-39bd-423b-a702-d21ad6573447") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:43.939495 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:43.939460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:45.037483 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.037440    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:45.037956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.037440    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:45.037956 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:45.037555    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7gjk" podUID="9b002bc5-39bd-423b-a702-d21ad6573447"
Sep 05 19:43:45.037956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.037454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:45.037956 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:45.037695    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:43:45.037956 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:45.037747    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xt85s" podUID="3a134963-6f04-4a07-ad64-8482ce725d59"
Sep 05 19:43:45.367046 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.367023    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-128.ec2.internal" event="NodeReady"
Sep 05 19:43:45.367211 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.367198    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 19:43:45.468495 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.468470    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wmxb8"]
Sep 05 19:43:45.485251 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.485222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.490145 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.490115    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 19:43:45.490303 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.490116    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\""
Sep 05 19:43:45.490451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.490433    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 19:43:45.491133 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.491112    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pzfw7"]
Sep 05 19:43:45.507390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.507363    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmxb8"]
Sep 05 19:43:45.507541 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.507483    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.510363 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.510340    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzfw7"]
Sep 05 19:43:45.512524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.512067    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7tzm\""
Sep 05 19:43:45.512524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.512164    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 19:43:45.512524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.512351    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 19:43:45.512524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.512427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 19:43:45.583446 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/905ab463-9ee2-45c9-ac7b-d3fd9f682065-config-volume\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.583581 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbz5\" (UniqueName: \"kubernetes.io/projected/24af6bd4-d01c-4e94-91d2-4d01efde2be2-kube-api-access-7nbz5\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.583581 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.583581 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ab463-9ee2-45c9-ac7b-d3fd9f682065-tmp-dir\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.583691 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kdj\" (UniqueName: \"kubernetes.io/projected/905ab463-9ee2-45c9-ac7b-d3fd9f682065-kube-api-access-c8kdj\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.583691 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.583637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24af6bd4-d01c-4e94-91d2-4d01efde2be2-cert\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.684749 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/905ab463-9ee2-45c9-ac7b-d3fd9f682065-config-volume\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.684922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684772    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbz5\" (UniqueName: \"kubernetes.io/projected/24af6bd4-d01c-4e94-91d2-4d01efde2be2-kube-api-access-7nbz5\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.684922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.684922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ab463-9ee2-45c9-ac7b-d3fd9f682065-tmp-dir\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.684922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kdj\" (UniqueName: \"kubernetes.io/projected/905ab463-9ee2-45c9-ac7b-d3fd9f682065-kube-api-access-c8kdj\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.684922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.684909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24af6bd4-d01c-4e94-91d2-4d01efde2be2-cert\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.685146 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:45.684924    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:45.685146 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:45.684986    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:46.184968203 +0000 UTC m=+35.602041608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:45.685283 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.685219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ab463-9ee2-45c9-ac7b-d3fd9f682065-tmp-dir\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.685353 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.685333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/905ab463-9ee2-45c9-ac7b-d3fd9f682065-config-volume\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.689312 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.689288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24af6bd4-d01c-4e94-91d2-4d01efde2be2-cert\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.697303 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.697274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kdj\" (UniqueName: \"kubernetes.io/projected/905ab463-9ee2-45c9-ac7b-d3fd9f682065-kube-api-access-c8kdj\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:45.697455 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.697439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbz5\" (UniqueName: \"kubernetes.io/projected/24af6bd4-d01c-4e94-91d2-4d01efde2be2-kube-api-access-7nbz5\") pod \"ingress-canary-pzfw7\" (UID: \"24af6bd4-d01c-4e94-91d2-4d01efde2be2\") " pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:45.816586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:45.816541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzfw7"
Sep 05 19:43:46.154308 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:46.153689    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzfw7"]
Sep 05 19:43:46.156398 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:43:46.156367    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af6bd4_d01c_4e94_91d2_4d01efde2be2.slice/crio-67e5e7401cc4a8897d320f7311cce0992c0bc0c7cf4c3b0079ab6a3be1b2300a WatchSource:0}: Error finding container 67e5e7401cc4a8897d320f7311cce0992c0bc0c7cf4c3b0079ab6a3be1b2300a: Status 404 returned error can't find the container with id 67e5e7401cc4a8897d320f7311cce0992c0bc0c7cf4c3b0079ab6a3be1b2300a
Sep 05 19:43:46.188572 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:46.188537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:46.188709 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:46.188657    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:46.188709 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:46.188704    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:47.188692051 +0000 UTC m=+36.605765466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:46.281207 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:46.279961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzfw7" event={"ID":"24af6bd4-d01c-4e94-91d2-4d01efde2be2","Type":"ContainerStarted","Data":"67e5e7401cc4a8897d320f7311cce0992c0bc0c7cf4c3b0079ab6a3be1b2300a"}
Sep 05 19:43:46.284526 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:46.284500    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="fcadbd6a27e1bb3c27ef96d836c14329a550f25edd2a10b2a4c8126f51636f00" exitCode=0
Sep 05 19:43:46.284671 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:46.284541    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"fcadbd6a27e1bb3c27ef96d836c14329a550f25edd2a10b2a4c8126f51636f00"}
Sep 05 19:43:47.037335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.037208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:43:47.037335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.037246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:43:47.037335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.037256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:43:47.040287 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.040260    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 19:43:47.040642 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.040619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\""
Sep 05 19:43:47.041872 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.041848    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 19:43:47.041872 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.041870    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 19:43:47.042020 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.041875    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\""
Sep 05 19:43:47.042020 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.041934    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 19:43:47.182775 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.182725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:47.196829 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.196799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:47.196960 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:47.196944    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:47.197053 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:47.197042    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:49.197025754 +0000 UTC m=+38.614099155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:47.285000 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.284966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:47.290880 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.290805    2575 generic.go:356] "Generic (PLEG): container finished" podID="f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f" containerID="2afa41b243eab42776a7693430f0cc46a93906a5db492681a8c4a35b954dfe27" exitCode=0
Sep 05 19:43:47.290880 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.290876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerDied","Data":"2afa41b243eab42776a7693430f0cc46a93906a5db492681a8c4a35b954dfe27"}
Sep 05 19:43:47.826933 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:47.826897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:48.298048 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:48.297961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" event={"ID":"f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f","Type":"ContainerStarted","Data":"c92ddaf4fd745136ac8b7ccbcafd31c465946033b13f20cbe269d44a221c0ece"}
Sep 05 19:43:48.325084 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:48.325032    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zb8q" podStartSLOduration=7.497808972 podStartE2EDuration="37.325014645s" totalImagesPullingTime="29.827205673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:48.324857929 +0000 UTC m=+37.741931353" watchObservedRunningTime="2026-09-05 19:43:48.325014645 +0000 UTC m=+37.742088069"
Sep 05 19:43:48.454936 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:48.454899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:48.555007 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:48.554933    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:49.210452 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:49.210227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:49.210452 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:49.210384    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:49.210637 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:49.210502    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:53.210485776 +0000 UTC m=+42.627559177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:49.301337 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:49.301299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzfw7" event={"ID":"24af6bd4-d01c-4e94-91d2-4d01efde2be2","Type":"ContainerStarted","Data":"e7f398e9f861ebb2a8760adf224885d7d0862a8cd02b2ca8422f529651534b70"}
Sep 05 19:43:49.321318 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:49.321263    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pzfw7" podStartSLOduration=1.587138159 podStartE2EDuration="4.321247541s" totalImagesPullingTime="2.734109382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:49.320819478 +0000 UTC m=+38.737892902" watchObservedRunningTime="2026-09-05 19:43:49.321247541 +0000 UTC m=+38.738320964"
Sep 05 19:43:49.971603 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:49.971572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:50.031420 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:50.031377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:53.237467 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:53.237434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:43:53.237865 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:53.237562    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:53.237865 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:43:53.237611    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:01.237598102 +0000 UTC m=+50.654671504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:54.954344 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:54.954309    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:55.052134 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:55.052103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:58.907295 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:58.907259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:58.967245 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:58.967219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:59.029845 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:59.029811    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:59.443988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:59.443952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:43:59.546576 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:43:59.546544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:00.067630 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:00.067600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:00.131852 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:00.131815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:01.298409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:01.298367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:44:01.298860 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:01.298520    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:01.298860 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:01.298591    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:17.298572596 +0000 UTC m=+66.715646008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:10.274793 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:10.274764    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbhxh"
Sep 05 19:44:10.426632 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:10.426597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:10.489337 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:10.489306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:10.554343 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:10.554261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:15.695738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.695696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:44:15.700821 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.700801    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 19:44:15.709107 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.709086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a134963-6f04-4a07-ad64-8482ce725d59-original-pull-secret\") pod \"global-pull-secret-syncer-xt85s\" (UID: \"3a134963-6f04-4a07-ad64-8482ce725d59\") " pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:44:15.796436 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.796401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:44:15.799939 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.799922    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 19:44:15.807210 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:15.807173    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 19:44:15.807262 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:15.807252    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:19.807234982 +0000 UTC m=+129.224308383 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : secret "metrics-daemon-secret" not found
Sep 05 19:44:15.856920 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.856897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xt85s"
Sep 05 19:44:15.897164 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.897138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:44:15.900372 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.900345    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 19:44:15.911004 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.910979    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 19:44:15.920492 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:15.920467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8m8\" (UniqueName: \"kubernetes.io/projected/9b002bc5-39bd-423b-a702-d21ad6573447-kube-api-access-vz8m8\") pod \"network-check-target-f7gjk\" (UID: \"9b002bc5-39bd-423b-a702-d21ad6573447\") " pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:44:16.153259 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:16.153221    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\""
Sep 05 19:44:16.160084 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:16.160051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xt85s"]
Sep 05 19:44:16.160084 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:16.160079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:44:16.162933 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:44:16.162906    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a134963_6f04_4a07_ad64_8482ce725d59.slice/crio-173bd70ad223bdd57fcf0f35dbd27660017105f6da158699eb3e49e3e950d2df WatchSource:0}: Error finding container 173bd70ad223bdd57fcf0f35dbd27660017105f6da158699eb3e49e3e950d2df: Status 404 returned error can't find the container with id 173bd70ad223bdd57fcf0f35dbd27660017105f6da158699eb3e49e3e950d2df
Sep 05 19:44:16.364391 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:16.364349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xt85s" event={"ID":"3a134963-6f04-4a07-ad64-8482ce725d59","Type":"ContainerStarted","Data":"173bd70ad223bdd57fcf0f35dbd27660017105f6da158699eb3e49e3e950d2df"}
Sep 05 19:44:16.458319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:16.458020    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7gjk"]
Sep 05 19:44:16.463321 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:44:16.463294    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b002bc5_39bd_423b_a702_d21ad6573447.slice/crio-b89e434d6030d250996f786dba4778112cfbf38b15b6838dddf914507a650f8a WatchSource:0}: Error finding container b89e434d6030d250996f786dba4778112cfbf38b15b6838dddf914507a650f8a: Status 404 returned error can't find the container with id b89e434d6030d250996f786dba4778112cfbf38b15b6838dddf914507a650f8a
Sep 05 19:44:17.308580 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:17.308533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:44:17.309126 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:17.308690    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:17.309126 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:17.308764    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:49.308742162 +0000 UTC m=+98.725815569 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:17.368785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:17.368747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7gjk" event={"ID":"9b002bc5-39bd-423b-a702-d21ad6573447","Type":"ContainerStarted","Data":"b89e434d6030d250996f786dba4778112cfbf38b15b6838dddf914507a650f8a"}
Sep 05 19:44:21.379707 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.379672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xt85s" event={"ID":"3a134963-6f04-4a07-ad64-8482ce725d59","Type":"ContainerStarted","Data":"2e0b0aa40d60d873a05732d15dfb15e1966a597eab595c748d33b4a40ffe4dd7"}
Sep 05 19:44:21.384252 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.384177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7gjk" event={"ID":"9b002bc5-39bd-423b-a702-d21ad6573447","Type":"ContainerStarted","Data":"296ee0652b099faf8f890ba59f153b5b909542baf19bebaaffdf28e3c082740b"}
Sep 05 19:44:21.384374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.384311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:44:21.399108 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.399059    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xt85s" podStartSLOduration=65.69941728 podStartE2EDuration="1m10.39904662s" totalImagesPullingTime="4.69962934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:21.398387729 +0000 UTC m=+70.815461153" watchObservedRunningTime="2026-09-05 19:44:21.39904662 +0000 UTC m=+70.816120043"
Sep 05 19:44:21.416376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.416338    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f7gjk" podStartSLOduration=66.020944701 podStartE2EDuration="1m10.416325075s" totalImagesPullingTime="4.395380374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:21.415321004 +0000 UTC m=+70.832394426" watchObservedRunningTime="2026-09-05 19:44:21.416325075 +0000 UTC m=+70.833398492"
Sep 05 19:44:21.885532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.885498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:21.945747 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:21.945714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:22.001994 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:22.001962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:27.401405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:27.401371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:27.462248 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:27.462213    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:44:49.400294 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:49.400257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:44:49.400725 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:49.400373    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:49.400725 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:44:49.400435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls podName:905ab463-9ee2-45c9-ac7b-d3fd9f682065 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:53.400420745 +0000 UTC m=+162.817494158 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls") pod "dns-default-wmxb8" (UID: "905ab463-9ee2-45c9-ac7b-d3fd9f682065") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:52.388404 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:44:52.388370    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f7gjk"
Sep 05 19:45:01.937290 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:01.937250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:19.813148 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:19.813089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:45:19.813669 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:19.813261    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 19:45:19.813669 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:19.813333    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs podName:40555d87-452a-46ac-a4ac-fba76c45fd18 nodeName:}" failed. No retries permitted until 2026-09-05 19:47:21.813319204 +0000 UTC m=+251.230392607 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs") pod "network-metrics-daemon-f82tc" (UID: "40555d87-452a-46ac-a4ac-fba76c45fd18") : secret "metrics-daemon-secret" not found
Sep 05 19:45:35.629168 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.629137    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hl7bh"]
Sep 05 19:45:35.630736 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.630720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.633160 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.633138    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sq8xm\""
Sep 05 19:45:35.633293 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.633281    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:45:35.633342 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.633295    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 05 19:45:35.634115 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.634100    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 05 19:45:35.634321 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.634303    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 05 19:45:35.636624 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.636604    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:45:35.638181 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.638163    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.638820 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.638802    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 05 19:45:35.640695 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.640676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 19:45:35.640781 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.640695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 19:45:35.640781 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.640711    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rgzgk\""
Sep 05 19:45:35.640781 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.640772    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 19:45:35.646508 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.646483    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hl7bh"]
Sep 05 19:45:35.647263 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.647249    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 19:45:35.657110 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.657092    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:45:35.720458 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720458 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj97\" (UniqueName: \"kubernetes.io/projected/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-kube-api-access-fqj97\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49lh\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720507    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-config\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-serving-cert\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.720970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.720970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-trusted-ca\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.720970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.720864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.801608 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.801570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:35.821499 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-trusted-ca\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.821618 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.821618 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.821720 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj97\" (UniqueName: \"kubernetes.io/projected/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-kube-api-access-fqj97\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.821720 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49lh\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.821819 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.821819 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-config\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.821990 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.822674 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.822231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.822674 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.822290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-serving-cert\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.822674 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.822343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.822674 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.822379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.823087 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.822785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-trusted-ca\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.823087 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.821885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.823214 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.823137    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.823288 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.823270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.823430 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:35.823405    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 19:45:35.823484 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:35.823434    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b67d4b7f6-bcw6f: secret "image-registry-tls" not found
Sep 05 19:45:35.823533 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:35.823514    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls podName:bf2477ac-7448-41e3-9c38-b263dcd7a91e nodeName:}" failed. No retries permitted until 2026-09-05 19:45:36.323485926 +0000 UTC m=+145.740559343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls") pod "image-registry-5b67d4b7f6-bcw6f" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e") : secret "image-registry-tls" not found
Sep 05 19:45:35.826832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.824620    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-config\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.826832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.824941    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.826832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.825258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-serving-cert\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.828366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.828340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.830583 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.830564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj97\" (UniqueName: \"kubernetes.io/projected/d07b9fe0-fd1d-447c-9a59-601164d0fcbc-kube-api-access-fqj97\") pod \"console-operator-85b454874b-hl7bh\" (UID: \"d07b9fe0-fd1d-447c-9a59-601164d0fcbc\") " pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:35.830658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.830610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.830794 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.830774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49lh\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:35.903241 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.903137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:35.939657 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:35.939627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:36.205810 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:36.205737    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hl7bh"]
Sep 05 19:45:36.210805 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:36.210584    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07b9fe0_fd1d_447c_9a59_601164d0fcbc.slice/crio-3d67b2f89af4d63430f82d42b20e6c07092eec2b724599312c2c27bfa87871a6 WatchSource:0}: Error finding container 3d67b2f89af4d63430f82d42b20e6c07092eec2b724599312c2c27bfa87871a6: Status 404 returned error can't find the container with id 3d67b2f89af4d63430f82d42b20e6c07092eec2b724599312c2c27bfa87871a6
Sep 05 19:45:36.327089 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:36.327052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:36.327283 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:36.327175    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 19:45:36.327283 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:36.327208    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b67d4b7f6-bcw6f: secret "image-registry-tls" not found
Sep 05 19:45:36.327283 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:36.327265    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls podName:bf2477ac-7448-41e3-9c38-b263dcd7a91e nodeName:}" failed. No retries permitted until 2026-09-05 19:45:37.327251732 +0000 UTC m=+146.744325146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls") pod "image-registry-5b67d4b7f6-bcw6f" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e") : secret "image-registry-tls" not found
Sep 05 19:45:36.572771 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:36.572735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" event={"ID":"d07b9fe0-fd1d-447c-9a59-601164d0fcbc","Type":"ContainerStarted","Data":"3d67b2f89af4d63430f82d42b20e6c07092eec2b724599312c2c27bfa87871a6"}
Sep 05 19:45:37.335473 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:37.335436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:37.335857 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:37.335622    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 19:45:37.335857 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:37.335647    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b67d4b7f6-bcw6f: secret "image-registry-tls" not found
Sep 05 19:45:37.335857 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:37.335713    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls podName:bf2477ac-7448-41e3-9c38-b263dcd7a91e nodeName:}" failed. No retries permitted until 2026-09-05 19:45:39.335692908 +0000 UTC m=+148.752766312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls") pod "image-registry-5b67d4b7f6-bcw6f" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e") : secret "image-registry-tls" not found
Sep 05 19:45:38.117098 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.117067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:38.177364 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.177329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:38.371946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.371912    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"]
Sep 05 19:45:38.374679 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.374663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.376851 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.376829    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 05 19:45:38.377260 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.377239    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:45:38.378034 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.378017    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xf6pq\""
Sep 05 19:45:38.378106 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.378041    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 05 19:45:38.378106 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.378062    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 05 19:45:38.384375 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.384354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"]
Sep 05 19:45:38.444164 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.444128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-config\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.444367 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.444173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-serving-cert\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.444367 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.444218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65pvt\" (UniqueName: \"kubernetes.io/projected/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-kube-api-access-65pvt\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.545437 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.545405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65pvt\" (UniqueName: \"kubernetes.io/projected/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-kube-api-access-65pvt\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.545613 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.545481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-config\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.545613 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.545508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-serving-cert\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.546043 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.546020    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-config\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.547676 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.547656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-serving-cert\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.554248 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.554223    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65pvt\" (UniqueName: \"kubernetes.io/projected/6a1baed6-6857-4eb5-9e70-552f51b0c3e3-kube-api-access-65pvt\") pod \"service-ca-operator-859f94c4b-xjfcg\" (UID: \"6a1baed6-6857-4eb5-9e70-552f51b0c3e3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.580231 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.580209    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/0.log"
Sep 05 19:45:38.580338 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.580248    2575 generic.go:356] "Generic (PLEG): container finished" podID="d07b9fe0-fd1d-447c-9a59-601164d0fcbc" containerID="daf132df9b3b029cb5a7fda431ac7a4acaf844d3098a275df9f378db3a593fcf" exitCode=255
Sep 05 19:45:38.580338 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.580293    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" event={"ID":"d07b9fe0-fd1d-447c-9a59-601164d0fcbc","Type":"ContainerDied","Data":"daf132df9b3b029cb5a7fda431ac7a4acaf844d3098a275df9f378db3a593fcf"}
Sep 05 19:45:38.580531 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.580516    2575 scope.go:117] "RemoveContainer" containerID="daf132df9b3b029cb5a7fda431ac7a4acaf844d3098a275df9f378db3a593fcf"
Sep 05 19:45:38.682878 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.682849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"
Sep 05 19:45:38.952622 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:38.952543    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg"]
Sep 05 19:45:38.955196 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:38.955146    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1baed6_6857_4eb5_9e70_552f51b0c3e3.slice/crio-af5f0ee00f565a5584189aaeb6b06a03bd4a855ed7be643f8e5a6d520ae35d2d WatchSource:0}: Error finding container af5f0ee00f565a5584189aaeb6b06a03bd4a855ed7be643f8e5a6d520ae35d2d: Status 404 returned error can't find the container with id af5f0ee00f565a5584189aaeb6b06a03bd4a855ed7be643f8e5a6d520ae35d2d
Sep 05 19:45:39.351942 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.351898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:39.352113 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.352059    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 19:45:39.352113 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.352079    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b67d4b7f6-bcw6f: secret "image-registry-tls" not found
Sep 05 19:45:39.352211 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.352146    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls podName:bf2477ac-7448-41e3-9c38-b263dcd7a91e nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.352129401 +0000 UTC m=+152.769202815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls") pod "image-registry-5b67d4b7f6-bcw6f" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e") : secret "image-registry-tls" not found
Sep 05 19:45:39.481509 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.481477    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-srp6j"]
Sep 05 19:45:39.484194 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.484159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j"
Sep 05 19:45:39.486755 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.486731    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dh8rd\""
Sep 05 19:45:39.493328 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.493275    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-srp6j"]
Sep 05 19:45:39.553482 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.553434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdz2p\" (UniqueName: \"kubernetes.io/projected/ffb33166-d8f4-42cb-a03c-ec0ebfb68023-kube-api-access-kdz2p\") pod \"network-check-source-685456655f-srp6j\" (UID: \"ffb33166-d8f4-42cb-a03c-ec0ebfb68023\") " pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j"
Sep 05 19:45:39.580309 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.580278    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p7d8x"]
Sep 05 19:45:39.583406 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.583383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.585221 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.585176    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:45:39.585785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.585680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/0.log"
Sep 05 19:45:39.585785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.585720    2575 generic.go:356] "Generic (PLEG): container finished" podID="d07b9fe0-fd1d-447c-9a59-601164d0fcbc" containerID="83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c" exitCode=255
Sep 05 19:45:39.585953 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.585804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" event={"ID":"d07b9fe0-fd1d-447c-9a59-601164d0fcbc","Type":"ContainerDied","Data":"83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c"}
Sep 05 19:45:39.585953 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.585853    2575 scope.go:117] "RemoveContainer" containerID="daf132df9b3b029cb5a7fda431ac7a4acaf844d3098a275df9f378db3a593fcf"
Sep 05 19:45:39.586658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.586060    2575 scope.go:117] "RemoveContainer" containerID="83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c"
Sep 05 19:45:39.586658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.586174    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 19:45:39.586658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.586262    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 19:45:39.586658 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.586270    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hl7bh_openshift-console-operator(d07b9fe0-fd1d-447c-9a59-601164d0fcbc)\"" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" podUID="d07b9fe0-fd1d-447c-9a59-601164d0fcbc"
Sep 05 19:45:39.586658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.586174    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 19:45:39.586957 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.586692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wszhj\""
Sep 05 19:45:39.587314 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.587130    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 19:45:39.587314 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.587288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:39.587955 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.587879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg" event={"ID":"6a1baed6-6857-4eb5-9e70-552f51b0c3e3","Type":"ContainerStarted","Data":"af5f0ee00f565a5584189aaeb6b06a03bd4a855ed7be643f8e5a6d520ae35d2d"}
Sep 05 19:45:39.594874 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.594809    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7d8x"]
Sep 05 19:45:39.645516 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.645436    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:39.654227 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdz2p\" (UniqueName: \"kubernetes.io/projected/ffb33166-d8f4-42cb-a03c-ec0ebfb68023-kube-api-access-kdz2p\") pod \"network-check-source-685456655f-srp6j\" (UID: \"ffb33166-d8f4-42cb-a03c-ec0ebfb68023\") " pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j"
Sep 05 19:45:39.654369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-crio-socket\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.654369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vz9j\" (UniqueName: \"kubernetes.io/projected/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-api-access-9vz9j\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.654493 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-data-volume\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.654622 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.654779 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.654758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.663503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.663477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdz2p\" (UniqueName: \"kubernetes.io/projected/ffb33166-d8f4-42cb-a03c-ec0ebfb68023-kube-api-access-kdz2p\") pod \"network-check-source-685456655f-srp6j\" (UID: \"ffb33166-d8f4-42cb-a03c-ec0ebfb68023\") " pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j"
Sep 05 19:45:39.755567 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-data-volume\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.755740 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.755740 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.755677    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:39.755740 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.755740 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.755738    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.255722383 +0000 UTC m=+149.672795784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:39.755919 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.755778    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:39.755919 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-crio-socket\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.755919 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:39.755842    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.255826768 +0000 UTC m=+149.672900170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:39.755919 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vz9j\" (UniqueName: \"kubernetes.io/projected/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-api-access-9vz9j\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.755919 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-crio-socket\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.756153 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.755921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-data-volume\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.764899 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.764877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vz9j\" (UniqueName: \"kubernetes.io/projected/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-api-access-9vz9j\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:39.794710 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:39.794677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j"
Sep 05 19:45:40.070446 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.070415    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-srp6j"]
Sep 05 19:45:40.073822 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:40.073791    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb33166_d8f4_42cb_a03c_ec0ebfb68023.slice/crio-302e5824b39c1f660db1bbeec94d09750d050a211e9c1d033703c4010acf5e86 WatchSource:0}: Error finding container 302e5824b39c1f660db1bbeec94d09750d050a211e9c1d033703c4010acf5e86: Status 404 returned error can't find the container with id 302e5824b39c1f660db1bbeec94d09750d050a211e9c1d033703c4010acf5e86
Sep 05 19:45:40.260704 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.260664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:40.260704 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.260703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:40.260892 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:40.260796    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:40.260892 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:40.260811    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:40.260892 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:40.260847    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.26083429 +0000 UTC m=+150.677907691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:40.260892 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:40.260860    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.260853953 +0000 UTC m=+150.677927354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:40.592680 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.592643    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:45:40.593098 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.593062    2575 scope.go:117] "RemoveContainer" containerID="83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c"
Sep 05 19:45:40.593288 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:40.593265    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hl7bh_openshift-console-operator(d07b9fe0-fd1d-447c-9a59-601164d0fcbc)\"" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" podUID="d07b9fe0-fd1d-447c-9a59-601164d0fcbc"
Sep 05 19:45:40.594424 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.594396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j" event={"ID":"ffb33166-d8f4-42cb-a03c-ec0ebfb68023","Type":"ContainerStarted","Data":"515cf6da04ed244ac934723de3b6c2caeebee1f501569b0b9619a2798c5a3c69"}
Sep 05 19:45:40.594531 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.594447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j" event={"ID":"ffb33166-d8f4-42cb-a03c-ec0ebfb68023","Type":"ContainerStarted","Data":"302e5824b39c1f660db1bbeec94d09750d050a211e9c1d033703c4010acf5e86"}
Sep 05 19:45:40.624043 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.623991    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-srp6j" podStartSLOduration=1.623972076 podStartE2EDuration="1.623972076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:40.623826575 +0000 UTC m=+150.040899999" watchObservedRunningTime="2026-09-05 19:45:40.623972076 +0000 UTC m=+150.041045512"
Sep 05 19:45:40.723601 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.723565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:40.755002 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.754965    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"]
Sep 05 19:45:40.759443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.759410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"
Sep 05 19:45:40.762080 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.762057    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 05 19:45:40.762222 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.762101    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-llmbg\""
Sep 05 19:45:40.762294 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.762245    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 05 19:45:40.766213 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.766163    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"]
Sep 05 19:45:40.823853 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.823747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:40.866291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.866177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv4q\" (UniqueName: \"kubernetes.io/projected/2ea818ff-b0f7-41e2-a5c0-563fc484c7b6-kube-api-access-glv4q\") pod \"migrator-fdbb8fdff-crmwc\" (UID: \"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"
Sep 05 19:45:40.923003 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.922974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:40.967526 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.967496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv4q\" (UniqueName: \"kubernetes.io/projected/2ea818ff-b0f7-41e2-a5c0-563fc484c7b6-kube-api-access-glv4q\") pod \"migrator-fdbb8fdff-crmwc\" (UID: \"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"
Sep 05 19:45:40.975738 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:40.975708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv4q\" (UniqueName: \"kubernetes.io/projected/2ea818ff-b0f7-41e2-a5c0-563fc484c7b6-kube-api-access-glv4q\") pod \"migrator-fdbb8fdff-crmwc\" (UID: \"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"
Sep 05 19:45:41.070516 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.070471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"
Sep 05 19:45:41.271006 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.270960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:41.271268 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.271065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:41.271268 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:41.271087    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:41.271268 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:41.271151    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:41.271268 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:41.271157    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.271139848 +0000 UTC m=+152.688213261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:41.271268 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:41.271204    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.271180884 +0000 UTC m=+152.688254285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:41.345687 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.345606    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc"]
Sep 05 19:45:41.348239 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:41.348210    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea818ff_b0f7_41e2_a5c0_563fc484c7b6.slice/crio-13a4a90eb02474170caf3a801fe789dfa699f9b87fcac96a2c4a7948b88e3be4 WatchSource:0}: Error finding container 13a4a90eb02474170caf3a801fe789dfa699f9b87fcac96a2c4a7948b88e3be4: Status 404 returned error can't find the container with id 13a4a90eb02474170caf3a801fe789dfa699f9b87fcac96a2c4a7948b88e3be4
Sep 05 19:45:41.598501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.598457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg" event={"ID":"6a1baed6-6857-4eb5-9e70-552f51b0c3e3","Type":"ContainerStarted","Data":"f0ee315380c357b8ff392a2ac4639faba7d280ced130a0eb5a4d9026a13895ff"}
Sep 05 19:45:41.599615 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.599589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc" event={"ID":"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6","Type":"ContainerStarted","Data":"13a4a90eb02474170caf3a801fe789dfa699f9b87fcac96a2c4a7948b88e3be4"}
Sep 05 19:45:41.615221 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.615156    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg" podStartSLOduration=1.67101587 podStartE2EDuration="3.615140221s" totalImagesPullingTime="1.944124351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:41.614551109 +0000 UTC m=+151.031624532" watchObservedRunningTime="2026-09-05 19:45:41.615140221 +0000 UTC m=+151.032213644"
Sep 05 19:45:41.737558 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:41.737523    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h7bb2_c70e9649-272e-43b4-a83e-730d12d26234/dns-node-resolver/0.log"
Sep 05 19:45:42.337469 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:42.337443    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzr5g_9a9507e7-ca6e-41ff-b451-a22c805f69e9/node-ca/0.log"
Sep 05 19:45:42.356373 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:42.356342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:42.418112 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:42.418071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:42.789137 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:42.789051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:43.064113 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.064080    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:43.164278 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.164245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:43.287659 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.287616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:43.287659 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.287658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:43.287850 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.287772    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:43.287850 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.287833    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.287819987 +0000 UTC m=+156.704893393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:43.287921 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.287772    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:43.287921 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.287905    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.287894607 +0000 UTC m=+156.704968008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:43.337429 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.337356    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzfw7_24af6bd4-d01c-4e94-91d2-4d01efde2be2/serve-healthcheck-canary/0.log"
Sep 05 19:45:43.388805 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.388773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:43.388950 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.388927    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 19:45:43.388989 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.388952    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b67d4b7f6-bcw6f: secret "image-registry-tls" not found
Sep 05 19:45:43.389027 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:43.389004    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls podName:bf2477ac-7448-41e3-9c38-b263dcd7a91e nodeName:}" failed. No retries permitted until 2026-09-05 19:45:51.388989889 +0000 UTC m=+160.806063291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls") pod "image-registry-5b67d4b7f6-bcw6f" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e") : secret "image-registry-tls" not found
Sep 05 19:45:43.608622 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.608521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc" event={"ID":"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6","Type":"ContainerStarted","Data":"b3ce1b83b6c09c6eec9b0bfbb750bd439c95a924d478b37df9c8ede4fcbeea84"}
Sep 05 19:45:43.608622 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.608580    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc" event={"ID":"2ea818ff-b0f7-41e2-a5c0-563fc484c7b6","Type":"ContainerStarted","Data":"d1b8daf23e98e49b4dbff849c766b6494e75b2d8d6fe757d3817617ffa1a4759"}
Sep 05 19:45:43.626734 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.626687    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-crmwc" podStartSLOduration=2.124616467 podStartE2EDuration="3.626670018s" totalImagesPullingTime="1.502053551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:43.625955183 +0000 UTC m=+153.043028610" watchObservedRunningTime="2026-09-05 19:45:43.626670018 +0000 UTC m=+153.043743440"
Sep 05 19:45:43.704717 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.704659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:43.770602 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:43.770551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:44.664666 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.664632    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7dc96"]
Sep 05 19:45:44.667541 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.667523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.670037 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.670012    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 05 19:45:44.670037 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.670024    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 05 19:45:44.670292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.670276    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 05 19:45:44.670349 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.670334    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 05 19:45:44.670410 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.670395    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-69lxx\""
Sep 05 19:45:44.679286 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.679262    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7dc96"]
Sep 05 19:45:44.698577 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.698546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:44.771598 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.771565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:44.798593 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.798556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztt9c\" (UniqueName: \"kubernetes.io/projected/e513de8b-8901-4b7a-a0d3-83f139f7c36f-kube-api-access-ztt9c\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.798727 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.798625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-cabundle\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.798769 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.798718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-key\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.899779 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.899728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztt9c\" (UniqueName: \"kubernetes.io/projected/e513de8b-8901-4b7a-a0d3-83f139f7c36f-kube-api-access-ztt9c\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.899779 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.899799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-cabundle\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.900051 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.899837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-key\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.900488 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.900458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-cabundle\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.902257 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.902238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e513de8b-8901-4b7a-a0d3-83f139f7c36f-signing-key\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.908388 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.908364    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztt9c\" (UniqueName: \"kubernetes.io/projected/e513de8b-8901-4b7a-a0d3-83f139f7c36f-kube-api-access-ztt9c\") pod \"service-ca-75d99c484-7dc96\" (UID: \"e513de8b-8901-4b7a-a0d3-83f139f7c36f\") " pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:44.980318 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:44.980214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7dc96"
Sep 05 19:45:45.247374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.247297    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7dc96"]
Sep 05 19:45:45.250017 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:45.249979    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode513de8b_8901_4b7a_a0d3_83f139f7c36f.slice/crio-771b099fbefb7ff1674181737458611ef35491ffba835eeff3126fc80329a6f8 WatchSource:0}: Error finding container 771b099fbefb7ff1674181737458611ef35491ffba835eeff3126fc80329a6f8: Status 404 returned error can't find the container with id 771b099fbefb7ff1674181737458611ef35491ffba835eeff3126fc80329a6f8
Sep 05 19:45:45.543341 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.543253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:45.617090 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.617047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7dc96" event={"ID":"e513de8b-8901-4b7a-a0d3-83f139f7c36f","Type":"ContainerStarted","Data":"bc7418f04dca545454836f3b5cb344ef443846c6ac6c0e36ccf4fa475de83124"}
Sep 05 19:45:45.617278 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.617097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7dc96" event={"ID":"e513de8b-8901-4b7a-a0d3-83f139f7c36f","Type":"ContainerStarted","Data":"771b099fbefb7ff1674181737458611ef35491ffba835eeff3126fc80329a6f8"}
Sep 05 19:45:45.635923 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.635869    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-7dc96" podStartSLOduration=1.635854269 podStartE2EDuration="1.635854269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:45.635036706 +0000 UTC m=+155.052110130" watchObservedRunningTime="2026-09-05 19:45:45.635854269 +0000 UTC m=+155.052927697"
Sep 05 19:45:45.940679 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.940636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:45.941067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.940710    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:45:45.941170 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:45.941156    2575 scope.go:117] "RemoveContainer" containerID="83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c"
Sep 05 19:45:45.941399 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:45.941375    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hl7bh_openshift-console-operator(d07b9fe0-fd1d-447c-9a59-601164d0fcbc)\"" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" podUID="d07b9fe0-fd1d-447c-9a59-601164d0fcbc"
Sep 05 19:45:46.839582 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:46.839547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:47.014930 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.014896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:47.215329 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.215241    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:47.300888 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.300834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:47.319153 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.319109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:47.319153 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.319162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:47.319419 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:47.319292    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:47.319419 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:47.319294    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:47.319419 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:47.319349    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.319332469 +0000 UTC m=+164.736405874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:47.319419 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:47.319389    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.319358092 +0000 UTC m=+164.736431498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:47.406015 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.405984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:47.536281 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:47.536155    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:48.036658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:48.036618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:48.329577 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:48.329540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:48.494525 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:48.494490    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wmxb8" podUID="905ab463-9ee2-45c9-ac7b-d3fd9f682065"
Sep 05 19:45:48.626528 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:48.626449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:45:48.935524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:48.935494    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:49.142082 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:49.142001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:49.334543 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:49.334505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:49.533535 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:49.533405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:50.062200 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:50.062153    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f82tc" podUID="40555d87-452a-46ac-a4ac-fba76c45fd18"
Sep 05 19:45:50.131372 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:50.131333    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:50.531707 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:50.531610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:50.729793 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:50.729759    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:50.933304 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:50.933262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:51.132061 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.132009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:51.333423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.333354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:51.456904 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.456870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:51.459524 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.459504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"image-registry-5b67d4b7f6-bcw6f\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") " pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:51.532627 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.532581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:51.547514 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.547488    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:51.821058 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:51.821021    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:45:51.823475 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:51.823447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2477ac_7448_41e3_9c38_b263dcd7a91e.slice/crio-db72a17c933919deb95b2d031df15b1ff92b273a2872bd85287528b9fb304eb4 WatchSource:0}: Error finding container db72a17c933919deb95b2d031df15b1ff92b273a2872bd85287528b9fb304eb4: Status 404 returned error can't find the container with id db72a17c933919deb95b2d031df15b1ff92b273a2872bd85287528b9fb304eb4
Sep 05 19:45:52.237362 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.237284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:52.640089 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.640058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" event={"ID":"bf2477ac-7448-41e3-9c38-b263dcd7a91e","Type":"ContainerStarted","Data":"868ab0dfa12b50b834d7bbe4643b03e62601cf63087b0feec2ba757fbdc2f1d0"}
Sep 05 19:45:52.640501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.640107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" event={"ID":"bf2477ac-7448-41e3-9c38-b263dcd7a91e","Type":"ContainerStarted","Data":"db72a17c933919deb95b2d031df15b1ff92b273a2872bd85287528b9fb304eb4"}
Sep 05 19:45:52.640501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.640199    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:45:52.660403 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.660356    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" podStartSLOduration=17.660340771 podStartE2EDuration="17.660340771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:52.659356009 +0000 UTC m=+162.076429431" watchObservedRunningTime="2026-09-05 19:45:52.660340771 +0000 UTC m=+162.077414194"
Sep 05 19:45:52.730886 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.730844    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:52.832666 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.832630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:52.932965 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:52.932867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:53.134502 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.134473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:53.327376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.327339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:53.476553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.476522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:45:53.478781 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.478762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/905ab463-9ee2-45c9-ac7b-d3fd9f682065-metrics-tls\") pod \"dns-default-wmxb8\" (UID: \"905ab463-9ee2-45c9-ac7b-d3fd9f682065\") " pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:45:53.533655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.533620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:53.730386 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.730296    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\""
Sep 05 19:45:53.737862 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:53.737841    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:45:54.013549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.013464    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmxb8"]
Sep 05 19:45:54.016233 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:45:54.016204    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905ab463_9ee2_45c9_ac7b_d3fd9f682065.slice/crio-f0b8bcee262912851338b10e054eb67ecabf68477a88ea8eb4de7a57885453e9 WatchSource:0}: Error finding container f0b8bcee262912851338b10e054eb67ecabf68477a88ea8eb4de7a57885453e9: Status 404 returned error can't find the container with id f0b8bcee262912851338b10e054eb67ecabf68477a88ea8eb4de7a57885453e9
Sep 05 19:45:54.327345 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.327316    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:54.508408 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.508343    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:54.568361 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.568322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:54.649086 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.649006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmxb8" event={"ID":"905ab463-9ee2-45c9-ac7b-d3fd9f682065","Type":"ContainerStarted","Data":"f0b8bcee262912851338b10e054eb67ecabf68477a88ea8eb4de7a57885453e9"}
Sep 05 19:45:54.929152 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:54.929058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:55.128085 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.128041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:55.330749 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.330700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:55.395251 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.395212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:55.395433 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.395263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:55.395493 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:55.395432    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:55.395544 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:45:55.395493    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls podName:5f2d3274-8edc-47da-b096-e65ecbb6a8e5 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:11.395474793 +0000 UTC m=+180.812548199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7d8x" (UID: "5f2d3274-8edc-47da-b096-e65ecbb6a8e5") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:55.395920 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.395897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:45:55.532795 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:55.532752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:56.336315 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.336275    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:56.657940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.657853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmxb8" event={"ID":"905ab463-9ee2-45c9-ac7b-d3fd9f682065","Type":"ContainerStarted","Data":"cf870928c94d6d59b8bbad748bc21c772b4f1d2263d677067a6ae99b8dc0d7ff"}
Sep 05 19:45:56.657940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.657938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmxb8" event={"ID":"905ab463-9ee2-45c9-ac7b-d3fd9f682065","Type":"ContainerStarted","Data":"58e5d1e24676f6e00f62b076fc1630beef1e3c9537e71b33ec9c5c181dcaefc5"}
Sep 05 19:45:56.658119 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.657996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:45:56.678409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.678359    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wmxb8" podStartSLOduration=129.97394176 podStartE2EDuration="2m11.678342945s" totalImagesPullingTime="1.704401185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:56.676649334 +0000 UTC m=+166.093722756" watchObservedRunningTime="2026-09-05 19:45:56.678342945 +0000 UTC m=+166.095416368"
Sep 05 19:45:56.736533 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:56.736496    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:57.129636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:57.129605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:57.330024 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:57.329988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:45:57.536479 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:45:57.536389    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:00.036868 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.036836    2575 scope.go:117] "RemoveContainer" containerID="83d08f47f3319e559aa84adb99ddc64a91446a1e8c8ca9a20cd3c35fe63f522c"
Sep 05 19:46:00.674945 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.674918    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:46:00.675154 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.674976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" event={"ID":"d07b9fe0-fd1d-447c-9a59-601164d0fcbc","Type":"ContainerStarted","Data":"2daedcde605b4d8c586c0b7fd684e16f7b12713d745f53fa537593106988b4ae"}
Sep 05 19:46:00.675262 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.675244    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:46:00.679926 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.679902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-hl7bh"
Sep 05 19:46:00.696879 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.696828    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-hl7bh" podStartSLOduration=23.91144317 podStartE2EDuration="25.696813058s" totalImagesPullingTime="1.785369888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:00.695728602 +0000 UTC m=+170.112802026" watchObservedRunningTime="2026-09-05 19:46:00.696813058 +0000 UTC m=+170.113886480"
Sep 05 19:46:00.770983 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.770949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:00.875599 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.875549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:00.941062 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:00.940980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:01.040701 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:01.040671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:46:01.292956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:01.292871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:01.452506 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:01.452465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:01.648590 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:01.648551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:01.707812 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:01.707780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:02.259654 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:02.259610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:02.458520 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:02.458479    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:02.523393 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:02.523293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:02.651239 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:02.651204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:02.752315 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:02.752286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:04.652873 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:04.652799    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:46:04.776848 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:04.776801    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:04.873965 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:04.873931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:06.664027 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:06.663999    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wmxb8"
Sep 05 19:46:08.091564 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:08.091530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:08.192340 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:08.192303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:09.671061 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:09.671017    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:09.772638 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:09.772602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:09.849333 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:09.849295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:10.133093 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:10.133059    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:11.417579 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:11.417539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:46:11.419904 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:11.419884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f2d3274-8edc-47da-b096-e65ecbb6a8e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7d8x\" (UID: \"5f2d3274-8edc-47da-b096-e65ecbb6a8e5\") " pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:46:11.698616 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:11.698540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wszhj\""
Sep 05 19:46:11.706491 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:11.706471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7d8x"
Sep 05 19:46:11.975480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:11.975400    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7d8x"]
Sep 05 19:46:11.978898 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:46:11.978861    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2d3274_8edc_47da_b096_e65ecbb6a8e5.slice/crio-aa4963245221961d8917721327da204856ec013d69a02fcd698427792fa027c7 WatchSource:0}: Error finding container aa4963245221961d8917721327da204856ec013d69a02fcd698427792fa027c7: Status 404 returned error can't find the container with id aa4963245221961d8917721327da204856ec013d69a02fcd698427792fa027c7
Sep 05 19:46:12.079078 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.079043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:12.136710 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.136672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:12.717753 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.717587    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7d8x" event={"ID":"5f2d3274-8edc-47da-b096-e65ecbb6a8e5","Type":"ContainerStarted","Data":"3da8b6a3e79820eac6ba2409c74b243063b3c229451da9cf2e7455d9ebd1a0b7"}
Sep 05 19:46:12.717753 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.717655    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7d8x" event={"ID":"5f2d3274-8edc-47da-b096-e65ecbb6a8e5","Type":"ContainerStarted","Data":"aa4963245221961d8917721327da204856ec013d69a02fcd698427792fa027c7"}
Sep 05 19:46:12.878339 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.878304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:12.944301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:12.944248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:13.104721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:13.104690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:13.161991 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:13.161956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:13.311295 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:13.311255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:13.723938 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:13.723905    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7d8x" event={"ID":"5f2d3274-8edc-47da-b096-e65ecbb6a8e5","Type":"ContainerStarted","Data":"14f45b8fef0bf7c0101a147f0473645dd1eca338515791c71e73b54de18dba65"}
Sep 05 19:46:14.289565 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:14.289524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:14.390530 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:14.390479    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:14.658085 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:14.658054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:46:14.729774 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:14.729738    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7d8x" event={"ID":"5f2d3274-8edc-47da-b096-e65ecbb6a8e5","Type":"ContainerStarted","Data":"45c6d83f2eed2dd9a2b2f9022d7e2305bf4eca20a78cd85c755cdd4073804803"}
Sep 05 19:46:14.747712 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:14.747666    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p7d8x" podStartSLOduration=33.49543464 podStartE2EDuration="35.747651396s" totalImagesPullingTime="2.252216756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:45:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:14.747300976 +0000 UTC m=+184.164374400" watchObservedRunningTime="2026-09-05 19:46:14.747651396 +0000 UTC m=+184.164724818"
Sep 05 19:46:15.240121 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:15.240084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:15.296069 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:15.296035    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:16.489587 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:16.489530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:16.552644 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:16.552610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:16.988261 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:16.988195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:17.045470 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:17.045438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:18.678096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:18.678063    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:18.753949 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:18.753904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:22.283451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:22.283412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:22.383836 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:22.383808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:23.299373 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:23.299341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:24.504067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:24.504030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:24.560546 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:24.560515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:24.813863 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:24.813832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:24.870353 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:24.870313    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.034959 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.034909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.126566 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.126475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.245156 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.245117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.310409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.310369    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v74ds"]
Sep 05 19:46:25.314721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.314701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.317477 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.317454    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 19:46:25.317783 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.317760    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 19:46:25.317921 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.317900    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 19:46:25.318643 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.318618    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 19:46:25.318748 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.318615    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 19:46:25.318748 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.318618    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4sdpt\""
Sep 05 19:46:25.318846 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.318628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 19:46:25.323272 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.323249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.402774 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.402684    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.427278 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427278 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-sys\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-wtmp\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-root\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-metrics-client-ca\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-tls\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lld9\" (UniqueName: \"kubernetes.io/projected/beaf5a30-7efb-4893-8164-d650ce0d64e0-kube-api-access-8lld9\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427481 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.427657 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.427522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-textfile\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.501724 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.501695    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.528687 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-wtmp\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-root\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-metrics-client-ca\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-tls\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lld9\" (UniqueName: \"kubernetes.io/projected/beaf5a30-7efb-4893-8164-d650ce0d64e0-kube-api-access-8lld9\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528807    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-root\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-textfile\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528860    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-wtmp\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.528934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-sys\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.529008    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/beaf5a30-7efb-4893-8164-d650ce0d64e0-sys\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.529064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-textfile\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.529756 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.529625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.530093 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.530058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beaf5a30-7efb-4893-8164-d650ce0d64e0-metrics-client-ca\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.531311 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.531279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-tls\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.531413 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.531328    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beaf5a30-7efb-4893-8164-d650ce0d64e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.537101 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.537071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lld9\" (UniqueName: \"kubernetes.io/projected/beaf5a30-7efb-4893-8164-d650ce0d64e0-kube-api-access-8lld9\") pod \"node-exporter-v74ds\" (UID: \"beaf5a30-7efb-4893-8164-d650ce0d64e0\") " pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.603225 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.603169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:25.624301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.624275    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v74ds"
Sep 05 19:46:25.634979 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:46:25.634945    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaf5a30_7efb_4893_8164_d650ce0d64e0.slice/crio-a6063b20d00b58325054be2cbc63eab4b44068c4656d30cbc695ec48567efc19 WatchSource:0}: Error finding container a6063b20d00b58325054be2cbc63eab4b44068c4656d30cbc695ec48567efc19: Status 404 returned error can't find the container with id a6063b20d00b58325054be2cbc63eab4b44068c4656d30cbc695ec48567efc19
Sep 05 19:46:25.766090 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:25.766014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v74ds" event={"ID":"beaf5a30-7efb-4893-8164-d650ce0d64e0","Type":"ContainerStarted","Data":"a6063b20d00b58325054be2cbc63eab4b44068c4656d30cbc695ec48567efc19"}
Sep 05 19:46:26.061171 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.061136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:26.163195 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.163156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:26.274558 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.274523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:26.403174 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.403095    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:46:26.407435 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.407408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.409473 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.409437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:26.411035 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411010    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 19:46:26.411503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 19:46:26.411721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411524    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 19:46:26.411721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 19:46:26.411721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411591    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 19:46:26.411721 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411486    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9zhhr\""
Sep 05 19:46:26.411940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411789    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 19:46:26.411940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 19:46:26.411940 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.411886    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 19:46:26.412173 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.412155    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 19:46:26.427041 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.427017    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:46:26.470168 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.470127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:26.536172 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtrq\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.536586 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.536578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637471 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637471 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtrq\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.637655 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.637621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638043 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638357 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638425 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638377    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.638579 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.638066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.639431 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.639403    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.640976 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.640817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.640976 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.640887    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.641139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.641116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.641355 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.641334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.641459 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.641440    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.641680 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.641664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.641858 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.641838    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.642303 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.642281    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.642610 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.642590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.645434 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.645410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtrq\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq\") pod \"alertmanager-main-0\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.718573 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.718537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:46:26.772692 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.772660    2575 generic.go:356] "Generic (PLEG): container finished" podID="beaf5a30-7efb-4893-8164-d650ce0d64e0" containerID="6235152d17fdf5721b790f34ae1aa3e3295975cc727b4bf25fb8a2c959df9b9f" exitCode=0
Sep 05 19:46:26.772864 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.772743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v74ds" event={"ID":"beaf5a30-7efb-4893-8164-d650ce0d64e0","Type":"ContainerDied","Data":"6235152d17fdf5721b790f34ae1aa3e3295975cc727b4bf25fb8a2c959df9b9f"}
Sep 05 19:46:26.998818 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:26.998738    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:46:27.001561 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:46:27.001533    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a09ca95_13dd_4c50_99e4_6f26f333742a.slice/crio-1313b9177d09a76355e2973c52794f64843d70714c2fd9b674b8d6a03ca463db WatchSource:0}: Error finding container 1313b9177d09a76355e2973c52794f64843d70714c2fd9b674b8d6a03ca463db: Status 404 returned error can't find the container with id 1313b9177d09a76355e2973c52794f64843d70714c2fd9b674b8d6a03ca463db
Sep 05 19:46:27.300469 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.300375    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:27.397221 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.397165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:27.779682 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.779643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v74ds" event={"ID":"beaf5a30-7efb-4893-8164-d650ce0d64e0","Type":"ContainerStarted","Data":"1806ce1b6deea3ccff3a5995a33cb7b2063393bdcd2130427541b61ec5327118"}
Sep 05 19:46:27.780107 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.779706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v74ds" event={"ID":"beaf5a30-7efb-4893-8164-d650ce0d64e0","Type":"ContainerStarted","Data":"8d4661c0947e20bfcc9d0860fda32267ea6cabe1732662da969c1bfe0a52a85b"}
Sep 05 19:46:27.781086 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.781061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"1313b9177d09a76355e2973c52794f64843d70714c2fd9b674b8d6a03ca463db"}
Sep 05 19:46:27.804426 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:27.804374    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v74ds" podStartSLOduration=1.796478164 podStartE2EDuration="2.804359702s" totalImagesPullingTime="946.698235ms" totalInitContainerRuntime="61.183303ms" isStatefulPod=true podCreationTimestamp="2026-09-05 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:27.802723311 +0000 UTC m=+197.219796735" watchObservedRunningTime="2026-09-05 19:46:27.804359702 +0000 UTC m=+197.221433163"
Sep 05 19:46:28.287614 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.287570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:28.336282 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.336195    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f97597988-wtqv4"]
Sep 05 19:46:28.340240 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.340215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.343137 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343060    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 05 19:46:28.343292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343154    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 05 19:46:28.343292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 05 19:46:28.343292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343212    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 05 19:46:28.343475 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343342    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 05 19:46:28.343624 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343604    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-blgkf\""
Sep 05 19:46:28.343624 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.343618    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8d3c0331s67t4\""
Sep 05 19:46:28.353101 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.353079    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f97597988-wtqv4"]
Sep 05 19:46:28.361675 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.361637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:28.454418 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-grpc-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rj5\" (UniqueName: \"kubernetes.io/projected/6ea72caf-299b-449b-b87f-fecf7c9c1084-kube-api-access-74rj5\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454650    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea72caf-299b-449b-b87f-fecf7c9c1084-metrics-client-ca\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.454832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.454708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555431 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555431 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea72caf-299b-449b-b87f-fecf7c9c1084-metrics-client-ca\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555840 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-grpc-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.555840 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.555723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rj5\" (UniqueName: \"kubernetes.io/projected/6ea72caf-299b-449b-b87f-fecf7c9c1084-kube-api-access-74rj5\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.556347 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.556322    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ea72caf-299b-449b-b87f-fecf7c9c1084-metrics-client-ca\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558500 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558620 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558514    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558766 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558814 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-grpc-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558861 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.558861 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.558829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ea72caf-299b-449b-b87f-fecf7c9c1084-secret-thanos-querier-tls\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.573797 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.573774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rj5\" (UniqueName: \"kubernetes.io/projected/6ea72caf-299b-449b-b87f-fecf7c9c1084-kube-api-access-74rj5\") pod \"thanos-querier-5f97597988-wtqv4\" (UID: \"6ea72caf-299b-449b-b87f-fecf7c9c1084\") " pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.649455 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.649364    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:28.704831 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.704797    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:28.786163 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.786127    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca" exitCode=0
Sep 05 19:46:28.786552 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.786225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"}
Sep 05 19:46:28.805495 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.805462    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:28.923130 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:28.923057    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f97597988-wtqv4"]
Sep 05 19:46:28.925994 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:46:28.925967    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea72caf_299b_449b_b87f_fecf7c9c1084.slice/crio-b1f24944f1c55594b2cb3f39b221d4ed45020e78196aa6eec7496bbd4b187f57 WatchSource:0}: Error finding container b1f24944f1c55594b2cb3f39b221d4ed45020e78196aa6eec7496bbd4b187f57: Status 404 returned error can't find the container with id b1f24944f1c55594b2cb3f39b221d4ed45020e78196aa6eec7496bbd4b187f57
Sep 05 19:46:29.673233 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:29.673163    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" podUID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" containerName="registry" containerID="cri-o://868ab0dfa12b50b834d7bbe4643b03e62601cf63087b0feec2ba757fbdc2f1d0" gracePeriod=30
Sep 05 19:46:29.793490 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:29.793370    2575 generic.go:356] "Generic (PLEG): container finished" podID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" containerID="868ab0dfa12b50b834d7bbe4643b03e62601cf63087b0feec2ba757fbdc2f1d0" exitCode=0
Sep 05 19:46:29.793490 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:29.793408    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" event={"ID":"bf2477ac-7448-41e3-9c38-b263dcd7a91e","Type":"ContainerDied","Data":"868ab0dfa12b50b834d7bbe4643b03e62601cf63087b0feec2ba757fbdc2f1d0"}
Sep 05 19:46:29.795097 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:29.795060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"b1f24944f1c55594b2cb3f39b221d4ed45020e78196aa6eec7496bbd4b187f57"}
Sep 05 19:46:29.940758 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:29.940693    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:46:30.060697 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.060647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069635    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069734    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069778    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069829    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069862    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r49lh\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.069927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted\") pod \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\" (UID: \"bf2477ac-7448-41e3-9c38-b263dcd7a91e\") "
Sep 05 19:46:30.070319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.070091    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:46:30.070873 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.070744    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:46:30.073232 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.073168    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:46:30.073232 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.073207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh" (OuterVolumeSpecName: "kube-api-access-r49lh") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "kube-api-access-r49lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:46:30.074362 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.074310    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:46:30.074792 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.074667    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:46:30.074792 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.074769    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:46:30.084444 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.084406    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bf2477ac-7448-41e3-9c38-b263dcd7a91e" (UID: "bf2477ac-7448-41e3-9c38-b263dcd7a91e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:46:30.131713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.131671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:30.170645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170602    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-image-registry-private-configuration\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170646    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170664    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf2477ac-7448-41e3-9c38-b263dcd7a91e-installation-pull-secrets\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170680    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49lh\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-kube-api-access-r49lh\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170694    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf2477ac-7448-41e3-9c38-b263dcd7a91e-ca-trust-extracted\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170707    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-trusted-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170720    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf2477ac-7448-41e3-9c38-b263dcd7a91e-bound-sa-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.170883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.170732    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf2477ac-7448-41e3-9c38-b263dcd7a91e-registry-certificates\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:46:30.724214 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.724104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:30.800212 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.800160    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"
Sep 05 19:46:30.800647 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.800208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b67d4b7f6-bcw6f" event={"ID":"bf2477ac-7448-41e3-9c38-b263dcd7a91e","Type":"ContainerDied","Data":"db72a17c933919deb95b2d031df15b1ff92b273a2872bd85287528b9fb304eb4"}
Sep 05 19:46:30.800647 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.800280    2575 scope.go:117] "RemoveContainer" containerID="868ab0dfa12b50b834d7bbe4643b03e62601cf63087b0feec2ba757fbdc2f1d0"
Sep 05 19:46:30.826371 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.826340    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:46:30.832495 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:30.832467    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b67d4b7f6-bcw6f"]
Sep 05 19:46:31.042619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.042586    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" path="/var/lib/kubelet/pods/bf2477ac-7448-41e3-9c38-b263dcd7a91e/volumes"
Sep 05 19:46:31.286266 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.286158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:31.293096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.293061    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:46:31.293423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.293407    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" containerName="registry"
Sep 05 19:46:31.293465 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.293426    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" containerName="registry"
Sep 05 19:46:31.293500 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.293484    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2477ac-7448-41e3-9c38-b263dcd7a91e" containerName="registry"
Sep 05 19:46:31.296604 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.296587    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.301172 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.301151    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 19:46:31.302133 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.302109    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 19:46:31.302392 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.302372    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 19:46:31.302791 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.302772    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 19:46:31.302931 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.302918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 19:46:31.302974 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.302923    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 19:46:31.303790 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.303768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wf7bw\""
Sep 05 19:46:31.303892 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.303828    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 19:46:31.306539 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.306522    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 19:46:31.309881 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.309863    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:46:31.382713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.382651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6n6c\" (UniqueName: \"kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383367    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.383405 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.383399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.384307 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.384278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:31.484703 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484703 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6n6c\" (UniqueName: \"kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.484956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.484876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.485600 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.485567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.485705 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.485615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.485785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.485723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.485842 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.485783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.487392 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.487370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.487466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.487448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.494451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.494432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6n6c\" (UniqueName: \"kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c\") pod \"console-5d767d5bff-5mrh5\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") " pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.600619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.600579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:31.606460 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.606429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:31.808407 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.808337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"cb27b74e595c31560b18351415bb7eb81be1947b9b873e8fed2619c387cbbcfd"}
Sep 05 19:46:31.808850 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.808483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"b70ece096f479e91ee68c2cedba696b848c882924c0b9df05bb223266e20515b"}
Sep 05 19:46:31.808850 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.808531    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"91065486d333c106862c7d35139f8e9ed6ffb0249a2b37daa63c4feb3af366cd"}
Sep 05 19:46:31.814162 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.814122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"}
Sep 05 19:46:31.814317 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.814226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"}
Sep 05 19:46:31.814317 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.814269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"}
Sep 05 19:46:31.814418 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.814316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"}
Sep 05 19:46:31.814418 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.814355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"}
Sep 05 19:46:31.882730 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:31.882640    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:46:31.887202 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:46:31.887152    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642ede3d_8397_4f21_8fb1_fe4dc397a2ed.slice/crio-a6606f8091ba33cbdd45f0250a3f83407e3ccd13d4c4a3c5317a37e66ba6b76a WatchSource:0}: Error finding container a6606f8091ba33cbdd45f0250a3f83407e3ccd13d4c4a3c5317a37e66ba6b76a: Status 404 returned error can't find the container with id a6606f8091ba33cbdd45f0250a3f83407e3ccd13d4c4a3c5317a37e66ba6b76a
Sep 05 19:46:32.822790 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.822753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerStarted","Data":"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"}
Sep 05 19:46:32.826167 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.826138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"2ffd87a6dd4bc5a35b80720581ff74044a065b289b4c19a2d17e448c63a8b309"}
Sep 05 19:46:32.826320 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.826227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"44451ec95cce77634c8d7a26eb7481f911275ba70c3f011c36dcdb25b0f9698d"}
Sep 05 19:46:32.826320 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.826263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" event={"ID":"6ea72caf-299b-449b-b87f-fecf7c9c1084","Type":"ContainerStarted","Data":"d84685d0632994504ad46e63199b4f4aae808e94be4d3686676efd19869c95ba"}
Sep 05 19:46:32.826320 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.826313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:32.827472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.827449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d767d5bff-5mrh5" event={"ID":"642ede3d-8397-4f21-8fb1-fe4dc397a2ed","Type":"ContainerStarted","Data":"a6606f8091ba33cbdd45f0250a3f83407e3ccd13d4c4a3c5317a37e66ba6b76a"}
Sep 05 19:46:32.864758 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.864700    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.326957007 podStartE2EDuration="6.86468316s" totalImagesPullingTime="3.52154038s" totalInitContainerRuntime="16.185773ms" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:32.862788272 +0000 UTC m=+202.279861708" watchObservedRunningTime="2026-09-05 19:46:32.86468316 +0000 UTC m=+202.281756617"
Sep 05 19:46:32.895007 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:32.894946    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4" podStartSLOduration=1.945401853 podStartE2EDuration="4.894927561s" totalImagesPullingTime="2.949525708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:32.893775088 +0000 UTC m=+202.310848534" watchObservedRunningTime="2026-09-05 19:46:32.894927561 +0000 UTC m=+202.312000983"
Sep 05 19:46:34.844959 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:34.844921    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d767d5bff-5mrh5" event={"ID":"642ede3d-8397-4f21-8fb1-fe4dc397a2ed","Type":"ContainerStarted","Data":"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"}
Sep 05 19:46:34.864654 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:34.864602    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d767d5bff-5mrh5" podStartSLOduration=1.273053946 podStartE2EDuration="3.864584653s" totalImagesPullingTime="2.591530707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:34.863557332 +0000 UTC m=+204.280630754" watchObservedRunningTime="2026-09-05 19:46:34.864584653 +0000 UTC m=+204.281658076"
Sep 05 19:46:34.892163 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:34.892133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:34.948015 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:34.947983    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:38.844479 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:38.844451    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f97597988-wtqv4"
Sep 05 19:46:39.140890 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:39.140807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:39.204930 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:39.204894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:39.502891 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:39.502807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:39.605603 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:39.605565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:41.607157 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:41.607112    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:41.607662 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:41.607199    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:41.611909 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:41.611884    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:41.876577 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:41.876502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:46:42.078737 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:42.078705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:42.137963 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:42.137878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:42.199548 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:42.199512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:46.896380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:46.896341    2575 generic.go:356] "Generic (PLEG): container finished" podID="6a1baed6-6857-4eb5-9e70-552f51b0c3e3" containerID="f0ee315380c357b8ff392a2ac4639faba7d280ced130a0eb5a4d9026a13895ff" exitCode=0
Sep 05 19:46:46.896755 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:46.896413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg" event={"ID":"6a1baed6-6857-4eb5-9e70-552f51b0c3e3","Type":"ContainerDied","Data":"f0ee315380c357b8ff392a2ac4639faba7d280ced130a0eb5a4d9026a13895ff"}
Sep 05 19:46:46.896755 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:46.896737    2575 scope.go:117] "RemoveContainer" containerID="f0ee315380c357b8ff392a2ac4639faba7d280ced130a0eb5a4d9026a13895ff"
Sep 05 19:46:47.902451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:47.902416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xjfcg" event={"ID":"6a1baed6-6857-4eb5-9e70-552f51b0c3e3","Type":"ContainerStarted","Data":"9c3d11f37868ad16464bbda5f9bb4d4fa1439bb0d4df07e2b7d79a90d13bfe27"}
Sep 05 19:46:48.746401 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:48.746369    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:48.843471 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:48.843440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:48.902519 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:48.902482    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:54.501430 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:54.501396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:46:54.561369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:46:54.561335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:09.508619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:09.508579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:09.569348 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:09.569310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:14.108732 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:14.108682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:14.165663 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:14.165628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:21.910553 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:21.910506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:47:21.912762 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:21.912741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40555d87-452a-46ac-a4ac-fba76c45fd18-metrics-certs\") pod \"network-metrics-daemon-f82tc\" (UID: \"40555d87-452a-46ac-a4ac-fba76c45fd18\") " pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:47:22.044534 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:22.044500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\""
Sep 05 19:47:22.052053 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:22.052030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f82tc"
Sep 05 19:47:22.325883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:22.325846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f82tc"]
Sep 05 19:47:22.328855 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:47:22.328822    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40555d87_452a_46ac_a4ac_fba76c45fd18.slice/crio-b064785659efa719c67769d25825f7568fbf63a4033b6f10a1199941859a5a02 WatchSource:0}: Error finding container b064785659efa719c67769d25825f7568fbf63a4033b6f10a1199941859a5a02: Status 404 returned error can't find the container with id b064785659efa719c67769d25825f7568fbf63a4033b6f10a1199941859a5a02
Sep 05 19:47:23.030561 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:23.030521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f82tc" event={"ID":"40555d87-452a-46ac-a4ac-fba76c45fd18","Type":"ContainerStarted","Data":"b064785659efa719c67769d25825f7568fbf63a4033b6f10a1199941859a5a02"}
Sep 05 19:47:24.036398 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:24.036358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f82tc" event={"ID":"40555d87-452a-46ac-a4ac-fba76c45fd18","Type":"ContainerStarted","Data":"e1e4895e5516eb82c8ddc4d87de7e3b68d495648e319b2e823b754a1051c6595"}
Sep 05 19:47:24.036796 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:24.036446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f82tc" event={"ID":"40555d87-452a-46ac-a4ac-fba76c45fd18","Type":"ContainerStarted","Data":"d3ca5c6226b9db4a93b97ec9aaf4fae0a3c4d83d9bb03530df2793fdf449d1d9"}
Sep 05 19:47:24.057923 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:24.057870    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f82tc" podStartSLOduration=252.143471095 podStartE2EDuration="4m13.057853072s" totalImagesPullingTime="914.381977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:24.05712543 +0000 UTC m=+253.474198852" watchObservedRunningTime="2026-09-05 19:47:24.057853072 +0000 UTC m=+253.474926550"
Sep 05 19:47:24.153676 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:24.153641    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:24.220341 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:24.220305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:34.558202 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:34.558097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:34.659053 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:34.659006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:44.608891 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:44.608845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:44.965728 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:44.965632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:45.701833 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.701802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:45.702235 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702105    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="alertmanager" containerID="cri-o://98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39" gracePeriod=120
Sep 05 19:47:45.702235 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702159    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-metric" containerID="cri-o://85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd" gracePeriod=120
Sep 05 19:47:45.702235 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702196    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-web" containerID="cri-o://fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943" gracePeriod=120
Sep 05 19:47:45.702423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702216    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="config-reloader" containerID="cri-o://98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b" gracePeriod=120
Sep 05 19:47:45.702423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702259    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="prom-label-proxy" containerID="cri-o://4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0" gracePeriod=120
Sep 05 19:47:45.702423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:45.702258    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy" containerID="cri-o://c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13" gracePeriod=120
Sep 05 19:47:46.120242 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120206    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0" exitCode=0
Sep 05 19:47:46.120242 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120235    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13" exitCode=0
Sep 05 19:47:46.120242 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120242    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b" exitCode=0
Sep 05 19:47:46.120242 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120248    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39" exitCode=0
Sep 05 19:47:46.120555 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120212    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"}
Sep 05 19:47:46.120555 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"}
Sep 05 19:47:46.120555 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"}
Sep 05 19:47:46.120555 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.120450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"}
Sep 05 19:47:46.270909 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.270871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:46.333688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.333653    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:46.946763 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:46.946740    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.015688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015604    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015671    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015724    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgtrq\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015803    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015830    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.015887 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015868    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.016202 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.016202 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.015948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.016342 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.016316    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:47:47.016522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.016476    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:47:47.016633 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.016612    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.016695 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.016654    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.016751 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.016695    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy\") pod \"7a09ca95-13dd-4c50-99e4-6f26f333742a\" (UID: \"7a09ca95-13dd-4c50-99e4-6f26f333742a\") "
Sep 05 19:47:47.017108 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.017084    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-metrics-client-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.017231 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.017114    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.017231 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.017085    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:47:47.019386 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.019360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:47:47.019486 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.019378    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.019785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.019748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.019987 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.019951    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq" (OuterVolumeSpecName: "kube-api-access-wgtrq") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "kube-api-access-wgtrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:47:47.020130 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.020096    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.020888 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.020849    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume" (OuterVolumeSpecName: "config-volume") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.020986 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.020909    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out" (OuterVolumeSpecName: "config-out") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:47:47.021058 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.021035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.023960 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.023925    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.029980 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.029953    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config" (OuterVolumeSpecName: "web-config") pod "7a09ca95-13dd-4c50-99e4-6f26f333742a" (UID: "7a09ca95-13dd-4c50-99e4-6f26f333742a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:47:47.117843 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117810    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-cluster-tls-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.117843 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117837    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-tls-assets\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.117843 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117845    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-volume\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.117843 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117854    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtrq\" (UniqueName: \"kubernetes.io/projected/7a09ca95-13dd-4c50-99e4-6f26f333742a-kube-api-access-wgtrq\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117864    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117874    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117883    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-web-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117891    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-config-out\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117898    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7a09ca95-13dd-4c50-99e4-6f26f333742a-alertmanager-main-db\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117907    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.118099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.117916    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7a09ca95-13dd-4c50-99e4-6f26f333742a-secret-alertmanager-main-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:47:47.127793 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127763    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd" exitCode=0
Sep 05 19:47:47.127793 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127790    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerID="fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943" exitCode=0
Sep 05 19:47:47.127962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"}
Sep 05 19:47:47.127962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"}
Sep 05 19:47:47.127962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127863    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.128060 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.127961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7a09ca95-13dd-4c50-99e4-6f26f333742a","Type":"ContainerDied","Data":"1313b9177d09a76355e2973c52794f64843d70714c2fd9b674b8d6a03ca463db"}
Sep 05 19:47:47.128060 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.128008    2575 scope.go:117] "RemoveContainer" containerID="4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"
Sep 05 19:47:47.135548 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.135527    2575 scope.go:117] "RemoveContainer" containerID="85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"
Sep 05 19:47:47.143630 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.143604    2575 scope.go:117] "RemoveContainer" containerID="c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"
Sep 05 19:47:47.151246 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.151223    2575 scope.go:117] "RemoveContainer" containerID="fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"
Sep 05 19:47:47.157737 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.157714    2575 scope.go:117] "RemoveContainer" containerID="98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"
Sep 05 19:47:47.159910 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.159889    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:47.164825 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.164716    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:47.164889 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.164841    2575 scope.go:117] "RemoveContainer" containerID="98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"
Sep 05 19:47:47.171521 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.171494    2575 scope.go:117] "RemoveContainer" containerID="8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"
Sep 05 19:47:47.177905 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.177888    2575 scope.go:117] "RemoveContainer" containerID="4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"
Sep 05 19:47:47.178178 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.178160    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0\": container with ID starting with 4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0 not found: ID does not exist" containerID="4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"
Sep 05 19:47:47.178238 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178202    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"} err="failed to get container status \"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0\": rpc error: code = NotFound desc = could not find container \"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0\": container with ID starting with 4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0 not found: ID does not exist"
Sep 05 19:47:47.178289 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178240    2575 scope.go:117] "RemoveContainer" containerID="85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"
Sep 05 19:47:47.178505 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.178486    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd\": container with ID starting with 85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd not found: ID does not exist" containerID="85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"
Sep 05 19:47:47.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178511    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"} err="failed to get container status \"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd\": rpc error: code = NotFound desc = could not find container \"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd\": container with ID starting with 85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd not found: ID does not exist"
Sep 05 19:47:47.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178529    2575 scope.go:117] "RemoveContainer" containerID="c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"
Sep 05 19:47:47.178788 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.178772    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13\": container with ID starting with c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13 not found: ID does not exist" containerID="c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"
Sep 05 19:47:47.178830 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178794    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"} err="failed to get container status \"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13\": rpc error: code = NotFound desc = could not find container \"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13\": container with ID starting with c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13 not found: ID does not exist"
Sep 05 19:47:47.178830 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.178808    2575 scope.go:117] "RemoveContainer" containerID="fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"
Sep 05 19:47:47.179039 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.179024    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943\": container with ID starting with fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943 not found: ID does not exist" containerID="fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"
Sep 05 19:47:47.179079 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179044    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"} err="failed to get container status \"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943\": rpc error: code = NotFound desc = could not find container \"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943\": container with ID starting with fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943 not found: ID does not exist"
Sep 05 19:47:47.179079 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179058    2575 scope.go:117] "RemoveContainer" containerID="98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"
Sep 05 19:47:47.179306 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.179289    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b\": container with ID starting with 98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b not found: ID does not exist" containerID="98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"
Sep 05 19:47:47.179355 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179311    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"} err="failed to get container status \"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b\": rpc error: code = NotFound desc = could not find container \"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b\": container with ID starting with 98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b not found: ID does not exist"
Sep 05 19:47:47.179355 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179324    2575 scope.go:117] "RemoveContainer" containerID="98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"
Sep 05 19:47:47.179542 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.179526    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39\": container with ID starting with 98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39 not found: ID does not exist" containerID="98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"
Sep 05 19:47:47.179582 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179547    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"} err="failed to get container status \"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39\": rpc error: code = NotFound desc = could not find container \"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39\": container with ID starting with 98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39 not found: ID does not exist"
Sep 05 19:47:47.179582 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179561    2575 scope.go:117] "RemoveContainer" containerID="8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"
Sep 05 19:47:47.179789 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:47:47.179775    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca\": container with ID starting with 8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca not found: ID does not exist" containerID="8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"
Sep 05 19:47:47.179832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179793    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"} err="failed to get container status \"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca\": rpc error: code = NotFound desc = could not find container \"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca\": container with ID starting with 8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca not found: ID does not exist"
Sep 05 19:47:47.179832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.179804    2575 scope.go:117] "RemoveContainer" containerID="4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"
Sep 05 19:47:47.180037 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180020    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0"} err="failed to get container status \"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0\": rpc error: code = NotFound desc = could not find container \"4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0\": container with ID starting with 4bd95400073ce71e12304dd4b387007e2edb92c3d420b60af134d5829dcee0e0 not found: ID does not exist"
Sep 05 19:47:47.180085 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180039    2575 scope.go:117] "RemoveContainer" containerID="85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"
Sep 05 19:47:47.180284 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180268    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd"} err="failed to get container status \"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd\": rpc error: code = NotFound desc = could not find container \"85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd\": container with ID starting with 85409fab729a34301b8913e6535bbe166c14c66446e321853255a2319b12a5cd not found: ID does not exist"
Sep 05 19:47:47.180335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180285    2575 scope.go:117] "RemoveContainer" containerID="c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"
Sep 05 19:47:47.180493 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180480    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13"} err="failed to get container status \"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13\": rpc error: code = NotFound desc = could not find container \"c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13\": container with ID starting with c4f5ee70a1ccf07b8f49e260b639630f37d65a3b8cd33e6e4aa35706d9d7ec13 not found: ID does not exist"
Sep 05 19:47:47.180532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180493    2575 scope.go:117] "RemoveContainer" containerID="fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"
Sep 05 19:47:47.180668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180655    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943"} err="failed to get container status \"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943\": rpc error: code = NotFound desc = could not find container \"fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943\": container with ID starting with fdc7ef32ccea4a08ddddb780798f57252903178b8de418e40a1bbc44dc15c943 not found: ID does not exist"
Sep 05 19:47:47.180668 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180667    2575 scope.go:117] "RemoveContainer" containerID="98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"
Sep 05 19:47:47.180864 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180850    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b"} err="failed to get container status \"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b\": rpc error: code = NotFound desc = could not find container \"98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b\": container with ID starting with 98e3a88472d29fa53f6561bf0b479cc2f87339a78642cd7016990ddeac4f8f1b not found: ID does not exist"
Sep 05 19:47:47.180864 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.180862    2575 scope.go:117] "RemoveContainer" containerID="98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"
Sep 05 19:47:47.181058 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.181040    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39"} err="failed to get container status \"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39\": rpc error: code = NotFound desc = could not find container \"98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39\": container with ID starting with 98bb9cff92360a7ac0cd12fe49b1283e0ef516dee096a404485a38a87d1c3e39 not found: ID does not exist"
Sep 05 19:47:47.181102 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.181059    2575 scope.go:117] "RemoveContainer" containerID="8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"
Sep 05 19:47:47.181301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.181287    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca"} err="failed to get container status \"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca\": rpc error: code = NotFound desc = could not find container \"8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca\": container with ID starting with 8c6146f79e0c84b5b6e9034c6af9682dd6a008b1136ef182dd036161d5189fca not found: ID does not exist"
Sep 05 19:47:47.198141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198120    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:47.198415 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198402    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="alertmanager"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198418    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="alertmanager"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198428    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="config-reloader"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198434    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="config-reloader"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198442    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198448    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198458    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-metric"
Sep 05 19:47:47.198466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-metric"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198474    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-web"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198479    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-web"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198486    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="init-config-reloader"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198491    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="init-config-reloader"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198497    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="prom-label-proxy"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198502    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="prom-label-proxy"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198536    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="prom-label-proxy"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198543    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="alertmanager"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198549    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="config-reloader"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198555    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198561    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-web"
Sep 05 19:47:47.198689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.198566    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" containerName="kube-rbac-proxy-metric"
Sep 05 19:47:47.203587 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.203572    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.211452 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.211435    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 19:47:47.212199 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 19:47:47.212312 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212291    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 19:47:47.212312 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212300    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 19:47:47.212464 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212292    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 19:47:47.212464 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212295    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9zhhr\""
Sep 05 19:47:47.212616 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.212603    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 19:47:47.213552 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.213535    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 19:47:47.213706 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.213692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 19:47:47.217809 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.217791    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 19:47:47.218670 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-config-volume\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218783 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218783 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218874 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-config-out\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218939 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.218992 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.219050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.218993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv6j2\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-kube-api-access-mv6j2\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.219050 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.219023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-web-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.219143 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.219048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.219143 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.219097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.219143 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.219122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.224012 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.223993    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:47.319975 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.319948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.319981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-config-volume\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320417 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-config-out\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv6j2\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-kube-api-access-mv6j2\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-web-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.320855 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.320756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.321256 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.321086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.321488 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.321460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.321930 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.321903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ed562f-5637-47bc-9156-612e4056d78c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.323291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.323389 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.323492 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.323570 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.323801 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.324374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.323940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-config-volume\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.324374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.324337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-web-config\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.324494 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.324426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68ed562f-5637-47bc-9156-612e4056d78c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.324531 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.324511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68ed562f-5637-47bc-9156-612e4056d78c-config-out\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.330029 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.330012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv6j2\" (UniqueName: \"kubernetes.io/projected/68ed562f-5637-47bc-9156-612e4056d78c-kube-api-access-mv6j2\") pod \"alertmanager-main-0\" (UID: \"68ed562f-5637-47bc-9156-612e4056d78c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.512087 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.512056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 19:47:47.794740 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:47.794709    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 19:47:47.795560 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:47:47.795530    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ed562f_5637_47bc_9156_612e4056d78c.slice/crio-dd2972519e9423ab397a56fb0dd57783d4dfdaf8c7b21d43ab25e08951a9cdf7 WatchSource:0}: Error finding container dd2972519e9423ab397a56fb0dd57783d4dfdaf8c7b21d43ab25e08951a9cdf7: Status 404 returned error can't find the container with id dd2972519e9423ab397a56fb0dd57783d4dfdaf8c7b21d43ab25e08951a9cdf7
Sep 05 19:47:48.133913 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:48.133874    2575 generic.go:356] "Generic (PLEG): container finished" podID="68ed562f-5637-47bc-9156-612e4056d78c" containerID="04817db5a9a020150974b4db0c880a958727f0724d17fea4b65dff9ec03d87e0" exitCode=0
Sep 05 19:47:48.134309 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:48.133939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerDied","Data":"04817db5a9a020150974b4db0c880a958727f0724d17fea4b65dff9ec03d87e0"}
Sep 05 19:47:48.134309 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:48.134008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"dd2972519e9423ab397a56fb0dd57783d4dfdaf8c7b21d43ab25e08951a9cdf7"}
Sep 05 19:47:49.045631 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.045550    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a09ca95-13dd-4c50-99e4-6f26f333742a" path="/var/lib/kubelet/pods/7a09ca95-13dd-4c50-99e4-6f26f333742a/volumes"
Sep 05 19:47:49.140921 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.140886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"f910f55ec06e22587a96370f38e2bc216fc2d02ebf6e24dcf292402a17cf3108"}
Sep 05 19:47:49.141306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.140951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"b712b4474a667d7445063a9467860385dcaf7e55b46b2db46e077c3e374ab05a"}
Sep 05 19:47:49.141306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.140976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"5286350c77e7c6eaf633a958efdd448d56b4d5d01ad179641140ffa181f71447"}
Sep 05 19:47:49.141306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.140999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"e371e5753c2adfcf129e9eba7ac42a00c9b53eb2f8f7de7d394a58a80ab07d4b"}
Sep 05 19:47:49.141306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.141025    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"73a7739163294d6cb95b28c77b57f2412d3ee24beed05cc51ff1c6f613bf52f2"}
Sep 05 19:47:49.141306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.141051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68ed562f-5637-47bc-9156-612e4056d78c","Type":"ContainerStarted","Data":"312befbe25e53d6313ccbe88f961363e8013a78c9ef128b643c7847c0e6e714c"}
Sep 05 19:47:49.196873 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.196808    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.180183308 podStartE2EDuration="2.196792199s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.608891ms" isStatefulPod=false podCreationTimestamp="2026-09-05 19:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:49.195526784 +0000 UTC m=+278.612600233" watchObservedRunningTime="2026-09-05 19:47:49.196792199 +0000 UTC m=+278.613865621"
Sep 05 19:47:49.714445 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.714406    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"]
Sep 05 19:47:49.719707 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.719684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.722589 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.722565    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 05 19:47:49.722883 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.722862    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 05 19:47:49.723196 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.723167    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 05 19:47:49.723496 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.723482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 05 19:47:49.724686 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.724665    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 05 19:47:49.725120 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.725100    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v64z5\""
Sep 05 19:47:49.730470 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.730448    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 05 19:47:49.740561 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.740525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.740709 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.740576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-metrics-client-ca\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.740955 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.740927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.741076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.740993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkz9\" (UniqueName: \"kubernetes.io/projected/907f9b51-1f22-42f9-9c35-2bb9142c0180-kube-api-access-frkz9\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.741076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.741034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-federate-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.741076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.741061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.741239 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.741102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.741239 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.741126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.742430 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.742410    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"]
Sep 05 19:47:49.842378 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842378 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-metrics-client-ca\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkz9\" (UniqueName: \"kubernetes.io/projected/907f9b51-1f22-42f9-9c35-2bb9142c0180-kube-api-access-frkz9\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-federate-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842844 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.842844 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.842680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.843636 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.843605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-metrics-client-ca\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.843767 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.843694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.843970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.843952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907f9b51-1f22-42f9-9c35-2bb9142c0180-serving-certs-ca-bundle\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.845104 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.845074    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-telemeter-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.845214 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.845154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.845264 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.845215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.845324 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.845309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/907f9b51-1f22-42f9-9c35-2bb9142c0180-federate-client-tls\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:49.852285 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:49.852253    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkz9\" (UniqueName: \"kubernetes.io/projected/907f9b51-1f22-42f9-9c35-2bb9142c0180-kube-api-access-frkz9\") pod \"telemeter-client-6f7b9b8f9f-j5bff\" (UID: \"907f9b51-1f22-42f9-9c35-2bb9142c0180\") " pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:50.029435 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:50.029346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"
Sep 05 19:47:50.318281 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:50.318150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff"]
Sep 05 19:47:51.150479 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:51.150431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff" event={"ID":"907f9b51-1f22-42f9-9c35-2bb9142c0180","Type":"ContainerStarted","Data":"856abc55331416438263223cd72136984e3d4fab7cc5a3b8d8a47a41c0c1b566"}
Sep 05 19:47:52.156423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.156381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff" event={"ID":"907f9b51-1f22-42f9-9c35-2bb9142c0180","Type":"ContainerStarted","Data":"66c56160a003e6516871e9466605b3f025239ae136a8791403fe9479b64fd430"}
Sep 05 19:47:52.156780 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.156454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff" event={"ID":"907f9b51-1f22-42f9-9c35-2bb9142c0180","Type":"ContainerStarted","Data":"3112fadd5ed95e62841f4c4355cf76959196b2e35c048b768f768b464536df17"}
Sep 05 19:47:52.156780 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.156483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff" event={"ID":"907f9b51-1f22-42f9-9c35-2bb9142c0180","Type":"ContainerStarted","Data":"f949f6b1217b547f2a407e085d75912d0aedb2d38eb85b8389e06848707866d8"}
Sep 05 19:47:52.182602 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.182494    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f7b9b8f9f-j5bff" podStartSLOduration=1.58445569 podStartE2EDuration="3.182478729s" totalImagesPullingTime="1.598023039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:52.181149823 +0000 UTC m=+281.598223258" watchObservedRunningTime="2026-09-05 19:47:52.182478729 +0000 UTC m=+281.599552151"
Sep 05 19:47:52.278549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.278515    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:47:52.281979 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.281958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.294045 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.294023    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:47:52.340388 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.340355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:52.362773 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.362734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.362924 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.362779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn87\" (UniqueName: \"kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.362924 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.362862    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.362924 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.362912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.363070 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.362963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.363070 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.363002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.363070 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.363054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.440699 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.440619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:47:52.463801 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463801 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.463970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.463936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn87\" (UniqueName: \"kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.464644 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.464611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.464751 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.464696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.464808 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.464749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.464808 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.464762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.466452 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.466431    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.466757 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.466737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.473976 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.473956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn87\" (UniqueName: \"kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87\") pod \"console-6795894db4-d4t4q\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") " pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.592305 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.592269    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:47:52.866087 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:52.865998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:47:52.868686 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:47:52.868657    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd2b822_f50f_441b_880b_517afbb7d76d.slice/crio-34a139eebc8018d0b4048c90e7d362215a06aacb555762c8fc63cc9234272ed0 WatchSource:0}: Error finding container 34a139eebc8018d0b4048c90e7d362215a06aacb555762c8fc63cc9234272ed0: Status 404 returned error can't find the container with id 34a139eebc8018d0b4048c90e7d362215a06aacb555762c8fc63cc9234272ed0
Sep 05 19:47:53.162690 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:53.162606    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6795894db4-d4t4q" event={"ID":"7bd2b822-f50f-441b-880b-517afbb7d76d","Type":"ContainerStarted","Data":"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"}
Sep 05 19:47:53.162690 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:53.162657    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6795894db4-d4t4q" event={"ID":"7bd2b822-f50f-441b-880b-517afbb7d76d","Type":"ContainerStarted","Data":"34a139eebc8018d0b4048c90e7d362215a06aacb555762c8fc63cc9234272ed0"}
Sep 05 19:47:53.183810 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:47:53.183750    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6795894db4-d4t4q" podStartSLOduration=1.183733636 podStartE2EDuration="1.183733636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:53.181735025 +0000 UTC m=+282.598808459" watchObservedRunningTime="2026-09-05 19:47:53.183733636 +0000 UTC m=+282.600807058"
Sep 05 19:48:02.592859 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:02.592824    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:48:02.593249 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:02.592891    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:48:02.597511 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:02.597490    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:48:03.204317 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:03.204283    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:48:03.258406 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:03.258373    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:48:03.364210 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:03.364162    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:03.427410 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:03.427373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:03.486504 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:03.486416    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:05.291872 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:05.291835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:05.355364 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:05.355325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:10.926724 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:10.926694    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:48:10.927667 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:10.927646    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:48:10.943936 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:10.943912    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 19:48:28.279224 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.279168    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d767d5bff-5mrh5" podUID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" containerName="console" containerID="cri-o://8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0" gracePeriod=15
Sep 05 19:48:28.522683 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.522659    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:48:28.574504 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574524    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574558    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574589    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6n6c\" (UniqueName: \"kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574619    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574645 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.574946 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.574656    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config\") pod \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\" (UID: \"642ede3d-8397-4f21-8fb1-fe4dc397a2ed\") "
Sep 05 19:48:28.575096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575045    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config" (OuterVolumeSpecName: "console-config") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:48:28.575096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575060    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:48:28.575223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575093    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:48:28.575223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575145    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:48:28.575343 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575324    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.575390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575350    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-trusted-ca-bundle\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.575390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575365    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-oauth-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.575390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.575378    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.576967 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.576936    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c" (OuterVolumeSpecName: "kube-api-access-f6n6c") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "kube-api-access-f6n6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:48:28.577127 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.577110    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:48:28.577171 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.577123    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "642ede3d-8397-4f21-8fb1-fe4dc397a2ed" (UID: "642ede3d-8397-4f21-8fb1-fe4dc397a2ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:48:28.676415 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.676380    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.676415 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.676410    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-console-oauth-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:28.676610 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:28.676424    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6n6c\" (UniqueName: \"kubernetes.io/projected/642ede3d-8397-4f21-8fb1-fe4dc397a2ed-kube-api-access-f6n6c\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:48:29.300922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.300829    2575 generic.go:356] "Generic (PLEG): container finished" podID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" containerID="8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0" exitCode=0
Sep 05 19:48:29.300922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.300875    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d767d5bff-5mrh5"
Sep 05 19:48:29.300922 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.300907    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d767d5bff-5mrh5" event={"ID":"642ede3d-8397-4f21-8fb1-fe4dc397a2ed","Type":"ContainerDied","Data":"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"}
Sep 05 19:48:29.301421 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.300948    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d767d5bff-5mrh5" event={"ID":"642ede3d-8397-4f21-8fb1-fe4dc397a2ed","Type":"ContainerDied","Data":"a6606f8091ba33cbdd45f0250a3f83407e3ccd13d4c4a3c5317a37e66ba6b76a"}
Sep 05 19:48:29.301421 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.300968    2575 scope.go:117] "RemoveContainer" containerID="8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"
Sep 05 19:48:29.309229 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.309201    2575 scope.go:117] "RemoveContainer" containerID="8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"
Sep 05 19:48:29.309456 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:48:29.309440    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0\": container with ID starting with 8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0 not found: ID does not exist" containerID="8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"
Sep 05 19:48:29.309512 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.309462    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0"} err="failed to get container status \"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0\": rpc error: code = NotFound desc = could not find container \"8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0\": container with ID starting with 8c6f373a259b7a50089b0fc3c9a51b8c954cfa33dff0cc505c22ad12b95bd1c0 not found: ID does not exist"
Sep 05 19:48:29.320019 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.319997    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:48:29.326450 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.326423    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d767d5bff-5mrh5"]
Sep 05 19:48:29.993018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:29.992978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:30.072538 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:30.072504    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:31.042006 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:31.041967    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" path="/var/lib/kubelet/pods/642ede3d-8397-4f21-8fb1-fe4dc397a2ed/volumes"
Sep 05 19:48:37.904622 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:37.904582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:48:59.946532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:48:59.946493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:00.013291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.013262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:00.977962 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.977924    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:49:00.978451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.978396    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" containerName="console"
Sep 05 19:49:00.978451 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.978415    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" containerName="console"
Sep 05 19:49:00.978562 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.978488    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="642ede3d-8397-4f21-8fb1-fe4dc397a2ed" containerName="console"
Sep 05 19:49:00.982540 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.982519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:00.992676 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:00.992654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:49:01.032086 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.032054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:01.131308 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131490 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131490 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131490 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131615 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131615 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.131615 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.131584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bxh\" (UniqueName: \"kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.132833 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.132800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:01.232513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bxh\" (UniqueName: \"kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232837 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232837 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232551    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.232837 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.232584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.233361 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.233330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.233482 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.233332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.233482 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.233330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.233482 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.233388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.235530 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.235508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.235617 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.235593    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.240717 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.240691    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bxh\" (UniqueName: \"kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh\") pod \"console-6cfddfdbf-4vm4j\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") " pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.293376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.293340    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:01.566293 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:01.566254    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:49:01.570324 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:49:01.570299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7bc316_3479_4adb_9254_fbcc82e76e8d.slice/crio-0d78dc40d5e4ab7ee8dcb6b97cf572d24822dfd8de61794dbed5deafc0976a5c WatchSource:0}: Error finding container 0d78dc40d5e4ab7ee8dcb6b97cf572d24822dfd8de61794dbed5deafc0976a5c: Status 404 returned error can't find the container with id 0d78dc40d5e4ab7ee8dcb6b97cf572d24822dfd8de61794dbed5deafc0976a5c
Sep 05 19:49:02.425505 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:02.425462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfddfdbf-4vm4j" event={"ID":"ad7bc316-3479-4adb-9254-fbcc82e76e8d","Type":"ContainerStarted","Data":"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"}
Sep 05 19:49:02.425902 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:02.425532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfddfdbf-4vm4j" event={"ID":"ad7bc316-3479-4adb-9254-fbcc82e76e8d","Type":"ContainerStarted","Data":"0d78dc40d5e4ab7ee8dcb6b97cf572d24822dfd8de61794dbed5deafc0976a5c"}
Sep 05 19:49:02.449140 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:02.449087    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cfddfdbf-4vm4j" podStartSLOduration=2.449071544 podStartE2EDuration="2.449071544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:49:02.447866538 +0000 UTC m=+351.864939976" watchObservedRunningTime="2026-09-05 19:49:02.449071544 +0000 UTC m=+351.866144967"
Sep 05 19:49:11.294561 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.294469    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:11.294561 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.294540    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:11.299200 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.299157    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:11.466132 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.466103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:49:11.515392 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.515358    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:49:11.625531 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.625494    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:11.683301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.683260    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:11.951381 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:11.951301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:12.006980 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:12.006947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:24.514250 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:24.514212    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:24.615006 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:24.614958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:25.875609 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:25.875577    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:25.969075 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:25.969031    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:26.636098 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:26.636053    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:36.539393 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.539329    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6795894db4-d4t4q" podUID="7bd2b822-f50f-441b-880b-517afbb7d76d" containerName="console" containerID="cri-o://18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1" gracePeriod=15
Sep 05 19:49:36.788033 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.788003    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:49:36.823014 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.822981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjn87\" (UniqueName: \"kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823061    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823090    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823133    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823173    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823223 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config\") pod \"7bd2b822-f50f-441b-880b-517afbb7d76d\" (UID: \"7bd2b822-f50f-441b-880b-517afbb7d76d\") "
Sep 05 19:49:36.823567 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823465    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config" (OuterVolumeSpecName: "console-config") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:49:36.823625 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823574    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:49:36.823625 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823507    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca" (OuterVolumeSpecName: "service-ca") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:49:36.823849 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.823823    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:49:36.825286 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.825256    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:49:36.825480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.825462    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87" (OuterVolumeSpecName: "kube-api-access-cjn87") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "kube-api-access-cjn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:49:36.825834 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.825818    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7bd2b822-f50f-441b-880b-517afbb7d76d" (UID: "7bd2b822-f50f-441b-880b-517afbb7d76d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:49:36.924747 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924706    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924747 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924741    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-trusted-ca-bundle\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924747 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924751    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-oauth-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924759    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd2b822-f50f-441b-880b-517afbb7d76d-console-oauth-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924770    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn87\" (UniqueName: \"kubernetes.io/projected/7bd2b822-f50f-441b-880b-517afbb7d76d-kube-api-access-cjn87\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924780    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:36.924988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:36.924788    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd2b822-f50f-441b-880b-517afbb7d76d-console-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:49:37.555562 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.555524    2575 generic.go:356] "Generic (PLEG): container finished" podID="7bd2b822-f50f-441b-880b-517afbb7d76d" containerID="18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1" exitCode=0
Sep 05 19:49:37.556034 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.555572    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6795894db4-d4t4q"
Sep 05 19:49:37.556034 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.555609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6795894db4-d4t4q" event={"ID":"7bd2b822-f50f-441b-880b-517afbb7d76d","Type":"ContainerDied","Data":"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"}
Sep 05 19:49:37.556034 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.555658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6795894db4-d4t4q" event={"ID":"7bd2b822-f50f-441b-880b-517afbb7d76d","Type":"ContainerDied","Data":"34a139eebc8018d0b4048c90e7d362215a06aacb555762c8fc63cc9234272ed0"}
Sep 05 19:49:37.556034 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.555682    2575 scope.go:117] "RemoveContainer" containerID="18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"
Sep 05 19:49:37.563578 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.563560    2575 scope.go:117] "RemoveContainer" containerID="18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"
Sep 05 19:49:37.563859 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:49:37.563840    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1\": container with ID starting with 18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1 not found: ID does not exist" containerID="18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"
Sep 05 19:49:37.563908 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.563869    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1"} err="failed to get container status \"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1\": rpc error: code = NotFound desc = could not find container \"18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1\": container with ID starting with 18a755663a28354f37f1e0e12d35a952b5db6b93d28eed9e4fadd9265708e2f1 not found: ID does not exist"
Sep 05 19:49:37.573627 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.573604    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:49:37.577688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:37.577664    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6795894db4-d4t4q"]
Sep 05 19:49:39.042575 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:39.042540    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd2b822-f50f-441b-880b-517afbb7d76d" path="/var/lib/kubelet/pods/7bd2b822-f50f-441b-880b-517afbb7d76d/volumes"
Sep 05 19:49:42.155606 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.155565    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"]
Sep 05 19:49:42.156107 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.156002    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd2b822-f50f-441b-880b-517afbb7d76d" containerName="console"
Sep 05 19:49:42.156107 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.156021    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd2b822-f50f-441b-880b-517afbb7d76d" containerName="console"
Sep 05 19:49:42.156107 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.156096    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd2b822-f50f-441b-880b-517afbb7d76d" containerName="console"
Sep 05 19:49:42.160267 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.160234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.163112 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.163093    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 19:49:42.164267 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.164244    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 19:49:42.164363 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.164251    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 05 19:49:42.164363 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.164289    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 19:49:42.167892 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.167871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"]
Sep 05 19:49:42.182594 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.182567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:42.266275 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.266234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c482d33-4101-42a7-a6a3-f0231bf0bb72-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.266275 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.266280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gwl\" (UniqueName: \"kubernetes.io/projected/7c482d33-4101-42a7-a6a3-f0231bf0bb72-kube-api-access-22gwl\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.272935 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.272904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:42.296064 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.296032    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"]
Sep 05 19:49:42.299374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.299357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.301991 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.301966    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 05 19:49:42.302272 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.302252    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 05 19:49:42.302361 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.302279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 05 19:49:42.302361 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.302279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 05 19:49:42.311245 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.311222    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"]
Sep 05 19:49:42.335108 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.335079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:49:42.367335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqx6\" (UniqueName: \"kubernetes.io/projected/fd1b392b-48b4-4f38-87c4-0727b22de790-kube-api-access-flqx6\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b392b-48b4-4f38-87c4-0727b22de790-additional-service-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c482d33-4101-42a7-a6a3-f0231bf0bb72-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gwl\" (UniqueName: \"kubernetes.io/projected/7c482d33-4101-42a7-a6a3-f0231bf0bb72-kube-api-access-22gwl\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.367480 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.367831 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.367497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.370140 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.370114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c482d33-4101-42a7-a6a3-f0231bf0bb72-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.376735 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.376714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gwl\" (UniqueName: \"kubernetes.io/projected/7c482d33-4101-42a7-a6a3-f0231bf0bb72-kube-api-access-22gwl\") pod \"managed-serviceaccount-addon-agent-5767bddb78-mmxcx\" (UID: \"7c482d33-4101-42a7-a6a3-f0231bf0bb72\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.468845 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqx6\" (UniqueName: \"kubernetes.io/projected/fd1b392b-48b4-4f38-87c4-0727b22de790-kube-api-access-flqx6\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.468845 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b392b-48b4-4f38-87c4-0727b22de790-additional-service-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.469076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.469076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.469076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.469076 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.468909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.469752 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.469719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1b392b-48b4-4f38-87c4-0727b22de790-additional-service-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.471482 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.471451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-ca\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.471584 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.471512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.471584 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.471535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.471584 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.471578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fd1b392b-48b4-4f38-87c4-0727b22de790-hub\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.477755 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.477736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqx6\" (UniqueName: \"kubernetes.io/projected/fd1b392b-48b4-4f38-87c4-0727b22de790-kube-api-access-flqx6\") pod \"cluster-proxy-proxy-agent-676f8ddc4-lff5j\" (UID: \"fd1b392b-48b4-4f38-87c4-0727b22de790\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.479623 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.479602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"
Sep 05 19:49:42.608503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.608473    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"
Sep 05 19:49:42.749259 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.749162    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx"]
Sep 05 19:49:42.752395 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:49:42.752370    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c482d33_4101_42a7_a6a3_f0231bf0bb72.slice/crio-5b2e40d50a8fb5a2669573fb9bdaca1244dfd06b3250bbea39e50d877a2d9689 WatchSource:0}: Error finding container 5b2e40d50a8fb5a2669573fb9bdaca1244dfd06b3250bbea39e50d877a2d9689: Status 404 returned error can't find the container with id 5b2e40d50a8fb5a2669573fb9bdaca1244dfd06b3250bbea39e50d877a2d9689
Sep 05 19:49:42.754150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.754135    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:49:42.877348 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:42.877263    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j"]
Sep 05 19:49:42.879885 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:49:42.879856    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1b392b_48b4_4f38_87c4_0727b22de790.slice/crio-eab016134dc63c151cd535b6f4f98b809221635ecdf1c1d4d1ab649046e38e6f WatchSource:0}: Error finding container eab016134dc63c151cd535b6f4f98b809221635ecdf1c1d4d1ab649046e38e6f: Status 404 returned error can't find the container with id eab016134dc63c151cd535b6f4f98b809221635ecdf1c1d4d1ab649046e38e6f
Sep 05 19:49:43.579826 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:43.579795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j" event={"ID":"fd1b392b-48b4-4f38-87c4-0727b22de790","Type":"ContainerStarted","Data":"eab016134dc63c151cd535b6f4f98b809221635ecdf1c1d4d1ab649046e38e6f"}
Sep 05 19:49:43.580965 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:43.580931    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx" event={"ID":"7c482d33-4101-42a7-a6a3-f0231bf0bb72","Type":"ContainerStarted","Data":"5b2e40d50a8fb5a2669573fb9bdaca1244dfd06b3250bbea39e50d877a2d9689"}
Sep 05 19:49:47.600198 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.600141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j" event={"ID":"fd1b392b-48b4-4f38-87c4-0727b22de790","Type":"ContainerStarted","Data":"9e82223af8bc5dae26440c8602f5d156738444a13cc1bddf0ee1937bb9131bc0"}
Sep 05 19:49:47.600686 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.600209    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j" event={"ID":"fd1b392b-48b4-4f38-87c4-0727b22de790","Type":"ContainerStarted","Data":"e6d5695b4c4390e4f68ac6a6f627604af2921994e851c081da17cdecef0631d7"}
Sep 05 19:49:47.600686 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.600228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j" event={"ID":"fd1b392b-48b4-4f38-87c4-0727b22de790","Type":"ContainerStarted","Data":"2e762d84c31b335487535eedfae6dae4df216f84acee449892d2a4993aad8b6b"}
Sep 05 19:49:47.601714 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.601691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx" event={"ID":"7c482d33-4101-42a7-a6a3-f0231bf0bb72","Type":"ContainerStarted","Data":"1ac74f56a5b54cde240c16426f2c9b4c3b416ae0579abe23765100baeba4e963"}
Sep 05 19:49:47.624170 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.624118    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-676f8ddc4-lff5j" podStartSLOduration=1.237745783 podStartE2EDuration="5.624101769s" totalImagesPullingTime="4.386355986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:49:47.62244815 +0000 UTC m=+397.039521573" watchObservedRunningTime="2026-09-05 19:49:47.624101769 +0000 UTC m=+397.041175191"
Sep 05 19:49:47.643172 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:49:47.643115    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5767bddb78-mmxcx" podStartSLOduration=1.139735942 podStartE2EDuration="5.643098733s" totalImagesPullingTime="4.503362791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:49:47.642304533 +0000 UTC m=+397.059377955" watchObservedRunningTime="2026-09-05 19:49:47.643098733 +0000 UTC m=+397.060172155"
Sep 05 19:50:12.090088 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:12.090046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:13.298306 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:13.298266    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:33.720294 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:33.720195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:34.660021 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:34.659988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:37.302376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:37.302342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:37.410206 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:37.410152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:39.384099 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:39.384059    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:39.491860 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:39.491815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:39.584158 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:39.584124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:41.772427 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:41.772392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:42.186443 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:42.186398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:44.301905 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:44.301865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:44.401526 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:44.401495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:49.374565 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:49.374519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:50.716340 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:50.716303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:50.814808 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:50.814770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:57.569625 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:57.569590    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:57.669221 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:57.669171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:50:59.559208 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:50:59.559168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.018130 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.018092    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.112925 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.112884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.639905 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.639875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.842018 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.841985    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.940342 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.940206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:11.983982 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.983948    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"]
Sep 05 19:51:11.987363 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.987345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:11.990218 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.990180    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 05 19:51:11.990352 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.990243    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 05 19:51:11.990352 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.990265    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ngx5g\""
Sep 05 19:51:11.990352 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.990179    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 05 19:51:11.990352 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:11.990204    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 05 19:51:12.005253 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.005232    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"]
Sep 05 19:51:12.080332 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.080294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.080332 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.080333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrg54\" (UniqueName: \"kubernetes.io/projected/4c7b57ab-b186-4fea-9a80-fb20df883193-kube-api-access-mrg54\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.080724 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.080361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.080724 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.080432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.080724 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.080460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.105202 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.105149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.181709 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.181673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.181709 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.181709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrg54\" (UniqueName: \"kubernetes.io/projected/4c7b57ab-b186-4fea-9a80-fb20df883193-kube-api-access-mrg54\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.182027 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.181741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.182027 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.181765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.182027 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.181789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.182214 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.182122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.183957 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.183931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7b57ab-b186-4fea-9a80-fb20df883193-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.184255 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.184234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.184255 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.184246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c7b57ab-b186-4fea-9a80-fb20df883193-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.197620 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.197566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrg54\" (UniqueName: \"kubernetes.io/projected/4c7b57ab-b186-4fea-9a80-fb20df883193-kube-api-access-mrg54\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-qbtrn\" (UID: \"4c7b57ab-b186-4fea-9a80-fb20df883193\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.216159 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.216128    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.297469 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.297357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:12.297469 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.297384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.362215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.362136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.463132 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.463045    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.561673 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.561642    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:12.575923 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.575893    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"]
Sep 05 19:51:12.580117 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:51:12.580091    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7b57ab_b186_4fea_9a80_fb20df883193.slice/crio-83fc4f88762f5ef4cea5dc16ccd3548711f41151a894560065971295792da70e WatchSource:0}: Error finding container 83fc4f88762f5ef4cea5dc16ccd3548711f41151a894560065971295792da70e: Status 404 returned error can't find the container with id 83fc4f88762f5ef4cea5dc16ccd3548711f41151a894560065971295792da70e
Sep 05 19:51:12.939637 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:12.939601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn" event={"ID":"4c7b57ab-b186-4fea-9a80-fb20df883193","Type":"ContainerStarted","Data":"83fc4f88762f5ef4cea5dc16ccd3548711f41151a894560065971295792da70e"}
Sep 05 19:51:13.081831 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:13.081793    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:15.958098 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:15.958063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn" event={"ID":"4c7b57ab-b186-4fea-9a80-fb20df883193","Type":"ContainerStarted","Data":"fde104ac54422aa1a337b358aa41e2485bd7e5f6d2295cd81ece32d181b8edd4"}
Sep 05 19:51:16.963933 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:16.963842    2575 generic.go:356] "Generic (PLEG): container finished" podID="4c7b57ab-b186-4fea-9a80-fb20df883193" containerID="fde104ac54422aa1a337b358aa41e2485bd7e5f6d2295cd81ece32d181b8edd4" exitCode=0
Sep 05 19:51:16.963933 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:16.963917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn" event={"ID":"4c7b57ab-b186-4fea-9a80-fb20df883193","Type":"ContainerDied","Data":"fde104ac54422aa1a337b358aa41e2485bd7e5f6d2295cd81ece32d181b8edd4"}
Sep 05 19:51:17.970798 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:17.970763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn" event={"ID":"4c7b57ab-b186-4fea-9a80-fb20df883193","Type":"ContainerStarted","Data":"102164a56ead5ed69915f0141000151a07eb55d617b7ae3548c38469028c1ebe"}
Sep 05 19:51:17.971279 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:17.970882    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:17.994289 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:17.994242    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn" podStartSLOduration=2.67136161 podStartE2EDuration="6.994228782s" totalImagesPullingTime="3.252381422s" totalInitContainerRuntime="1.07048575s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:17.992302573 +0000 UTC m=+487.409375998" watchObservedRunningTime="2026-09-05 19:51:17.994228782 +0000 UTC m=+487.411302273"
Sep 05 19:51:21.574117 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.574084    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f6498b86-g2tkf"]
Sep 05 19:51:21.583847 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.583824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.587575 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.587546    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f6498b86-g2tkf"]
Sep 05 19:51:21.622230 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.622205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:21.667009 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.666980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-service-ca\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-oauth-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-oauth-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-trusted-ca-bundle\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667333 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.667333 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.667273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phsd\" (UniqueName: \"kubernetes.io/projected/a74f0648-b125-4547-a0f7-7c8dae9492ca-kube-api-access-9phsd\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.682229 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.682195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:21.768078 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phsd\" (UniqueName: \"kubernetes.io/projected/a74f0648-b125-4547-a0f7-7c8dae9492ca-kube-api-access-9phsd\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768302 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-service-ca\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768302 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-oauth-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-oauth-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-trusted-ca-bundle\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.768423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.769005 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.768981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-service-ca\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.769165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.769140    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.769291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.769110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-oauth-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.769291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.769175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0648-b125-4547-a0f7-7c8dae9492ca-trusted-ca-bundle\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.770758 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.770738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-serving-cert\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.770841 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.770794    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a74f0648-b125-4547-a0f7-7c8dae9492ca-console-oauth-config\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.775908 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.775879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phsd\" (UniqueName: \"kubernetes.io/projected/a74f0648-b125-4547-a0f7-7c8dae9492ca-kube-api-access-9phsd\") pod \"console-54f6498b86-g2tkf\" (UID: \"a74f0648-b125-4547-a0f7-7c8dae9492ca\") " pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:21.893988 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:21.893900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:22.171079 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.171056    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f6498b86-g2tkf"]
Sep 05 19:51:22.173569 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:51:22.173540    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74f0648_b125_4547_a0f7_7c8dae9492ca.slice/crio-6665951a3af0ce980da6498b7e19d73e65eafefe46eb452895c15a9b7cca20c3 WatchSource:0}: Error finding container 6665951a3af0ce980da6498b7e19d73e65eafefe46eb452895c15a9b7cca20c3: Status 404 returned error can't find the container with id 6665951a3af0ce980da6498b7e19d73e65eafefe46eb452895c15a9b7cca20c3
Sep 05 19:51:22.536053 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.535961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:22.598773 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.598738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:22.992017 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.991980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:22.993671 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.993638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f6498b86-g2tkf" event={"ID":"a74f0648-b125-4547-a0f7-7c8dae9492ca","Type":"ContainerStarted","Data":"e03d64aba53d5bb101c86aef30f9e10c96fe89c1df94c4f49268d7c72df15cdb"}
Sep 05 19:51:22.993797 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:22.993693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f6498b86-g2tkf" event={"ID":"a74f0648-b125-4547-a0f7-7c8dae9492ca","Type":"ContainerStarted","Data":"6665951a3af0ce980da6498b7e19d73e65eafefe46eb452895c15a9b7cca20c3"}
Sep 05 19:51:23.016149 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:23.016102    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54f6498b86-g2tkf" podStartSLOduration=2.016086884 podStartE2EDuration="2.016086884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:23.015958486 +0000 UTC m=+492.433031909" watchObservedRunningTime="2026-09-05 19:51:23.016086884 +0000 UTC m=+492.433160307"
Sep 05 19:51:27.333974 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:27.333936    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:27.390739 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:27.390701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:28.977293 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:28.977263    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-qbtrn"
Sep 05 19:51:29.360897 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.360862    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"]
Sep 05 19:51:29.368389 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.368367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.371364 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371325    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 05 19:51:29.371543 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 05 19:51:29.371625 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371578    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-grxmh\""
Sep 05 19:51:29.371689 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 05 19:51:29.371745 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 05 19:51:29.372029 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.371941    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:51:29.374513 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.374490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"]
Sep 05 19:51:29.538893 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.538852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5x9m\" (UniqueName: \"kubernetes.io/projected/f2c9221f-d221-49ce-ad3d-aecb7ea01145-kube-api-access-p5x9m\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.539066 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.538950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2c9221f-d221-49ce-ad3d-aecb7ea01145-manager-config\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.539066 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.539056    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.539150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.539085    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-metrics-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.640549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.640464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2c9221f-d221-49ce-ad3d-aecb7ea01145-manager-config\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.640549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.640521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.640549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.640536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-metrics-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.640800 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.640565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5x9m\" (UniqueName: \"kubernetes.io/projected/f2c9221f-d221-49ce-ad3d-aecb7ea01145-kube-api-access-p5x9m\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.641199 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.641146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2c9221f-d221-49ce-ad3d-aecb7ea01145-manager-config\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.642941 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.642917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-metrics-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.643036 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.642978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2c9221f-d221-49ce-ad3d-aecb7ea01145-cert\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.649534 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.649513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5x9m\" (UniqueName: \"kubernetes.io/projected/f2c9221f-d221-49ce-ad3d-aecb7ea01145-kube-api-access-p5x9m\") pod \"lws-controller-manager-6b9f469976-l2zbd\" (UID: \"f2c9221f-d221-49ce-ad3d-aecb7ea01145\") " pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.680484 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.680447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:29.951148 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:29.950992    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"]
Sep 05 19:51:29.953979 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:51:29.953954    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c9221f_d221_49ce_ad3d_aecb7ea01145.slice/crio-d6b0961b662ad15d5cb71a466c13bcd6c56ef5d2ac8ba553e5abe129ea87eeb7 WatchSource:0}: Error finding container d6b0961b662ad15d5cb71a466c13bcd6c56ef5d2ac8ba553e5abe129ea87eeb7: Status 404 returned error can't find the container with id d6b0961b662ad15d5cb71a466c13bcd6c56ef5d2ac8ba553e5abe129ea87eeb7
Sep 05 19:51:30.025514 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:30.025472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd" event={"ID":"f2c9221f-d221-49ce-ad3d-aecb7ea01145","Type":"ContainerStarted","Data":"d6b0961b662ad15d5cb71a466c13bcd6c56ef5d2ac8ba553e5abe129ea87eeb7"}
Sep 05 19:51:31.664491 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:31.664442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:31.766298 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:31.766237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:31.894301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:31.894236    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:31.894301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:31.894308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:31.900755 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:31.900726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:32.042832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.042738    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f6498b86-g2tkf"
Sep 05 19:51:32.120737 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.120698    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:32.142871 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.141687    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:51:32.342060 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.342025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:32.436028 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.435973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:32.549603 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.549565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:32.634088 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:32.633983    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:33.044063 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:33.043970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd" event={"ID":"f2c9221f-d221-49ce-ad3d-aecb7ea01145","Type":"ContainerStarted","Data":"e1822cb6d96b1783996af83a8fccdb3ff9621679ef3d82e290874877e15ea28c"}
Sep 05 19:51:33.044516 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:33.044165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:33.072203 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:33.072147    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd" podStartSLOduration=1.318571386 podStartE2EDuration="4.072131002s" totalImagesPullingTime="2.753559616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:33.071112719 +0000 UTC m=+502.488186143" watchObservedRunningTime="2026-09-05 19:51:33.072131002 +0000 UTC m=+502.489204425"
Sep 05 19:51:33.727669 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:33.727631    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:33.824591 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:33.824549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:34.782344 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:34.782308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:38.226429 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:38.226388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:38.326853 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:38.326821    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:38.516391 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:38.516309    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:38.609523 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:38.609484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:38.724381 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:38.724348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:42.753503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.753469    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xlm5x"]
Sep 05 19:51:42.756909 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.756889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.759249 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.759213    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 05 19:51:42.759378 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.759304    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r9rmc\""
Sep 05 19:51:42.766674 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.766647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xlm5x"]
Sep 05 19:51:42.782633 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.782591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:42.849380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.849348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:42.862046 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.862020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1542feed-728f-463c-a706-3e4b5e82c240-cert\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.862212 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.862069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plnx4\" (UniqueName: \"kubernetes.io/projected/1542feed-728f-463c-a706-3e4b5e82c240-kube-api-access-plnx4\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.938468 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.938437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:42.962602 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.962569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plnx4\" (UniqueName: \"kubernetes.io/projected/1542feed-728f-463c-a706-3e4b5e82c240-kube-api-access-plnx4\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.962743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.962651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1542feed-728f-463c-a706-3e4b5e82c240-cert\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.965089 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.965065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1542feed-728f-463c-a706-3e4b5e82c240-cert\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:42.972169 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:42.972142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plnx4\" (UniqueName: \"kubernetes.io/projected/1542feed-728f-463c-a706-3e4b5e82c240-kube-api-access-plnx4\") pod \"odh-model-controller-7998c67dc4-xlm5x\" (UID: \"1542feed-728f-463c-a706-3e4b5e82c240\") " pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:43.040718 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.040617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:43.066947 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.066918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:43.343764 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.343732    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xlm5x"]
Sep 05 19:51:43.347001 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:51:43.346971    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1542feed_728f_463c_a706_3e4b5e82c240.slice/crio-5274a7113583cfcfdc34cdaab0cafa15f2d6c6827315feab12ff56c0403c9f4d WatchSource:0}: Error finding container 5274a7113583cfcfdc34cdaab0cafa15f2d6c6827315feab12ff56c0403c9f4d: Status 404 returned error can't find the container with id 5274a7113583cfcfdc34cdaab0cafa15f2d6c6827315feab12ff56c0403c9f4d
Sep 05 19:51:43.408533 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.408486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:43.418112 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.418084    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cjj8p"]
Sep 05 19:51:43.422533 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.422515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.424956 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.424938    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 05 19:51:43.425086 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.425062    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-k2df8\""
Sep 05 19:51:43.436243 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.436215    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cjj8p"]
Sep 05 19:51:43.468260 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.468221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwml\" (UniqueName: \"kubernetes.io/projected/0cb46996-1381-4b32-9c28-49a2bf161710-kube-api-access-jxwml\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.468432 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.468343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb46996-1381-4b32-9c28-49a2bf161710-tls-certs\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.570548 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.570511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwml\" (UniqueName: \"kubernetes.io/projected/0cb46996-1381-4b32-9c28-49a2bf161710-kube-api-access-jxwml\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.570733 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.570612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb46996-1381-4b32-9c28-49a2bf161710-tls-certs\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.573440 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.573414    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb46996-1381-4b32-9c28-49a2bf161710-tls-certs\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.580298 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.580271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwml\" (UniqueName: \"kubernetes.io/projected/0cb46996-1381-4b32-9c28-49a2bf161710-kube-api-access-jxwml\") pod \"model-serving-api-54d89bd5fc-cjj8p\" (UID: \"0cb46996-1381-4b32-9c28-49a2bf161710\") " pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:43.604928 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.604849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:43.733220 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:43.733166    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:44.015773 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:44.015662    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cjj8p"]
Sep 05 19:51:44.051234 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:44.051214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6b9f469976-l2zbd"
Sep 05 19:51:44.097427 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:44.097338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x" event={"ID":"1542feed-728f-463c-a706-3e4b5e82c240","Type":"ContainerStarted","Data":"5274a7113583cfcfdc34cdaab0cafa15f2d6c6827315feab12ff56c0403c9f4d"}
Sep 05 19:51:44.100324 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:44.100286    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p" event={"ID":"0cb46996-1381-4b32-9c28-49a2bf161710","Type":"ContainerStarted","Data":"4bce23981acda1f38bcb42cc264cd9e0b9628384b9be3355477f157cb0514905"}
Sep 05 19:51:45.464242 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:45.464180    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:45.564015 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:45.563977    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:48.127932 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.127891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x" event={"ID":"1542feed-728f-463c-a706-3e4b5e82c240","Type":"ContainerStarted","Data":"82c2f2d1ff82495c96319b79b6166a14cb8b20dddf192eb9f0f8d55538ab19e4"}
Sep 05 19:51:48.128380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.127956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:48.129488 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.129463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p" event={"ID":"0cb46996-1381-4b32-9c28-49a2bf161710","Type":"ContainerStarted","Data":"5c875187293b5509a619ce8d04f931ea2ad8fd59e6a7de9cf5fb67000c685ebf"}
Sep 05 19:51:48.129736 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.129609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:51:48.146959 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.146902    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x" podStartSLOduration=2.216122964 podStartE2EDuration="6.146882365s" totalImagesPullingTime="3.930759401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:48.144561308 +0000 UTC m=+517.561634733" watchObservedRunningTime="2026-09-05 19:51:48.146882365 +0000 UTC m=+517.563955790"
Sep 05 19:51:48.163796 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:48.163741    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p" podStartSLOduration=1.8731641589999999 podStartE2EDuration="5.163727991s" totalImagesPullingTime="3.290563832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:48.161582395 +0000 UTC m=+517.578655821" watchObservedRunningTime="2026-09-05 19:51:48.163727991 +0000 UTC m=+517.580801415"
Sep 05 19:51:50.208929 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.208891    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"]
Sep 05 19:51:50.212336 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.212304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.216145 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216122    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 05 19:51:50.216322 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216212    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 05 19:51:50.216322 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216262    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 19:51:50.216471 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216265    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 05 19:51:50.216542 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216463    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 05 19:51:50.216542 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216495    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pxwq9\""
Sep 05 19:51:50.216542 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.216473    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 19:51:50.224861 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.224838    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"]
Sep 05 19:51:50.335680 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335835 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de3aa5-6e75-48cb-9e65-358b8245f35f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335835 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335835 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335941 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335941 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.335941 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.335898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftm22\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-kube-api-access-ftm22\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.436996 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.436945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437181 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de3aa5-6e75-48cb-9e65-358b8245f35f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437181 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437181 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437383 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437383 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437383 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftm22\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-kube-api-access-ftm22\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.437807 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.437778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.439770 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.439740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de3aa5-6e75-48cb-9e65-358b8245f35f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.439955 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.439932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.440003 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.439984    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.440046 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.440010    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.446945 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.446925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftm22\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-kube-api-access-ftm22\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.447049 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.447031    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de3aa5-6e75-48cb-9e65-358b8245f35f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ss2lh\" (UID: \"21de3aa5-6e75-48cb-9e65-358b8245f35f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.522532 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.522451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:50.808149 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:50.808124    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"]
Sep 05 19:51:50.810168 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:51:50.810133    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21de3aa5_6e75_48cb_9e65_358b8245f35f.slice/crio-c3a25291b5061cab04091e318837060673d8639bb4215185d79b586571847786 WatchSource:0}: Error finding container c3a25291b5061cab04091e318837060673d8639bb4215185d79b586571847786: Status 404 returned error can't find the container with id c3a25291b5061cab04091e318837060673d8639bb4215185d79b586571847786
Sep 05 19:51:51.143602 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:51.143520    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh" event={"ID":"21de3aa5-6e75-48cb-9e65-358b8245f35f","Type":"ContainerStarted","Data":"c3a25291b5061cab04091e318837060673d8639bb4215185d79b586571847786"}
Sep 05 19:51:53.108860 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.108821    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:51:53.109138 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.108889    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:51:53.482250 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.482149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:53.584456 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.584001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:53.894478 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.894441    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:53.958683 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:53.958644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:54.162219 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.162121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh" event={"ID":"21de3aa5-6e75-48cb-9e65-358b8245f35f","Type":"ContainerStarted","Data":"c466a98f7fee4b45fb9b58040745716beddd5a767a2bcc06288a01440224b7d4"}
Sep 05 19:51:54.162609 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.162425    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:54.164088 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.164065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh"
Sep 05 19:51:54.186554 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.186507    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ss2lh" podStartSLOduration=1.890215089 podStartE2EDuration="4.186489856s" totalImagesPullingTime="2.296274767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:54.184988097 +0000 UTC m=+523.602061519" watchObservedRunningTime="2026-09-05 19:51:54.186489856 +0000 UTC m=+523.603563278"
Sep 05 19:51:54.820992 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.820956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:54.905989 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:54.905946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:55.005895 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:55.005863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:51:57.167021 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.166983    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cfddfdbf-4vm4j" podUID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" containerName="console" containerID="cri-o://a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32" gracePeriod=15
Sep 05 19:51:57.414637 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.414613    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:51:57.516096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516053    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516096 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516077    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516110    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44bxh\" (UniqueName: \"kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516222    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516281    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516409 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516353    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config\") pod \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\" (UID: \"ad7bc316-3479-4adb-9254-fbcc82e76e8d\") "
Sep 05 19:51:57.516598 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516537    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:51:57.516598 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:51:57.516598 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516542    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:51:57.516743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516670    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.516816 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516798    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-trusted-ca-bundle\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.516876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516824    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-oauth-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.516876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.516859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config" (OuterVolumeSpecName: "console-config") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:51:57.518309 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.518285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh" (OuterVolumeSpecName: "kube-api-access-44bxh") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "kube-api-access-44bxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:51:57.518309 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.518293    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:51:57.518505 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.518380    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad7bc316-3479-4adb-9254-fbcc82e76e8d" (UID: "ad7bc316-3479-4adb-9254-fbcc82e76e8d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:51:57.617936 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.617902    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.617936 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.617930    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-serving-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.617936 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.617940    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bxh\" (UniqueName: \"kubernetes.io/projected/ad7bc316-3479-4adb-9254-fbcc82e76e8d-kube-api-access-44bxh\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:57.618174 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:57.617952    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7bc316-3479-4adb-9254-fbcc82e76e8d-console-oauth-config\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 19:51:58.182501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.182465    2575 generic.go:356] "Generic (PLEG): container finished" podID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" containerID="a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32" exitCode=0
Sep 05 19:51:58.182911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.182510    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfddfdbf-4vm4j"
Sep 05 19:51:58.182911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.182534    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfddfdbf-4vm4j" event={"ID":"ad7bc316-3479-4adb-9254-fbcc82e76e8d","Type":"ContainerDied","Data":"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"}
Sep 05 19:51:58.182911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.182572    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfddfdbf-4vm4j" event={"ID":"ad7bc316-3479-4adb-9254-fbcc82e76e8d","Type":"ContainerDied","Data":"0d78dc40d5e4ab7ee8dcb6b97cf572d24822dfd8de61794dbed5deafc0976a5c"}
Sep 05 19:51:58.182911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.182593    2575 scope.go:117] "RemoveContainer" containerID="a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"
Sep 05 19:51:58.190993 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.190975    2575 scope.go:117] "RemoveContainer" containerID="a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"
Sep 05 19:51:58.191259 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:51:58.191238    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32\": container with ID starting with a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32 not found: ID does not exist" containerID="a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"
Sep 05 19:51:58.191348 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.191266    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32"} err="failed to get container status \"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32\": rpc error: code = NotFound desc = could not find container \"a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32\": container with ID starting with a784f77af9e031c9d7d169dd3328c556d068b957b40b09325fec1379f3747c32 not found: ID does not exist"
Sep 05 19:51:58.206022 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.205999    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:51:58.211759 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:58.211737    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cfddfdbf-4vm4j"]
Sep 05 19:51:59.042287 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:59.042256    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" path="/var/lib/kubelet/pods/ad7bc316-3479-4adb-9254-fbcc82e76e8d/volumes"
Sep 05 19:51:59.137271 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:59.137243    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xlm5x"
Sep 05 19:51:59.139624 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:51:59.139599    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-cjj8p"
Sep 05 19:52:00.360002 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:00.359957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:00.421178 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:00.421141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:43.789131 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:43.789087    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:43.887158 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:43.887115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:45.821284 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:45.821253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:51.021955 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:51.021923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:52.973527 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:52.973489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:53.033713 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:53.033673    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:57.397604 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:57.397567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:52:59.800426 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:52:59.800391    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:00.242419 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:00.242373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:02.440010 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:02.439968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:02.542766 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:02.542708    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:02.639838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:02.639768    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:03.887774 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:03.887744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:04.599960 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:04.599928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:05.006179 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:05.006089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:05.241489 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:05.241452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:05.402686 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:05.402652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:06.076022 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:06.075981    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:06.173640 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:06.173604    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:10.809390 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:10.809354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:10.913366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:10.913332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:10.962498 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:10.962466    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:53:10.963008 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:10.962980    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:53:14.429590 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:14.429553    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:14.525776 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:14.525740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:14.631368 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:14.631329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:14.726604 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:14.726523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:16.879575 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:16.879541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:16.979756 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:16.979722    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:17.617848 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:17.617800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:17.859163 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:17.859132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:23.344367 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:23.344305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:23.441536 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:23.441504    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:23.541388 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:23.541344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:30.019794 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:30.019753    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:30.099884 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:30.099851    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:33.779601 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:33.779507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:33.880348 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:33.880303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:33.984356 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:33.984326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:34.039535 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:34.039448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:37.580743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:37.580706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:37.643359 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:37.643323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:43.341365 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:43.341331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:43.438335 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:43.438298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:43.541767 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:43.541726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:50.709585 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:50.709546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:50.780065 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:50.780027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:57.483945 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:57.483905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:53:57.854001 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:53:57.853966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:00.772115 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:00.772070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:00.871967 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:00.871935    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:11.934485 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:11.934444    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:12.038141 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.038099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:12.141072 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.141039    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:12.273892 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.273816    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:54:12.274150 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.274137    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" containerName="console"
Sep 05 19:54:12.274219 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.274151    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" containerName="console"
Sep 05 19:54:12.274267 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.274239    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7bc316-3479-4adb-9254-fbcc82e76e8d" containerName="console"
Sep 05 19:54:12.277089 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.277070    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.279817 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.279795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 05 19:54:12.279817 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.279811    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 05 19:54:12.279974 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.279816    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 05 19:54:12.280815 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.280794    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9t9m\""
Sep 05 19:54:12.289121 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.289095    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:54:12.308667 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.308639    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:54:12.376549 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.376502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdph\" (UniqueName: \"kubernetes.io/projected/3c8ee640-d75c-44cb-9948-06b0c62ece6b-kube-api-access-lbdph\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.376730 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.376571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c8ee640-d75c-44cb-9948-06b0c62ece6b-config-file\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.477692 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.477650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdph\" (UniqueName: \"kubernetes.io/projected/3c8ee640-d75c-44cb-9948-06b0c62ece6b-kube-api-access-lbdph\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.477876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.477713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c8ee640-d75c-44cb-9948-06b0c62ece6b-config-file\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.478400 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.478379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c8ee640-d75c-44cb-9948-06b0c62ece6b-config-file\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.487979 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.487953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdph\" (UniqueName: \"kubernetes.io/projected/3c8ee640-d75c-44cb-9948-06b0c62ece6b-kube-api-access-lbdph\") pod \"limitador-limitador-85d9db4d5d-tgslb\" (UID: \"3c8ee640-d75c-44cb-9948-06b0c62ece6b\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.587408 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.587374    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:12.763838 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.763802    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:12.861065 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.861034    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:54:12.863956 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:54:12.863930    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8ee640_d75c_44cb_9948_06b0c62ece6b.slice/crio-25692a6cb954346df29672a11c139dc77ad02ee96efd98dd6db6205016c44e71 WatchSource:0}: Error finding container 25692a6cb954346df29672a11c139dc77ad02ee96efd98dd6db6205016c44e71: Status 404 returned error can't find the container with id 25692a6cb954346df29672a11c139dc77ad02ee96efd98dd6db6205016c44e71
Sep 05 19:54:12.866019 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:12.865978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:13.794102 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:13.794062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb" event={"ID":"3c8ee640-d75c-44cb-9948-06b0c62ece6b","Type":"ContainerStarted","Data":"25692a6cb954346df29672a11c139dc77ad02ee96efd98dd6db6205016c44e71"}
Sep 05 19:54:15.805036 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:15.804999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb" event={"ID":"3c8ee640-d75c-44cb-9948-06b0c62ece6b","Type":"ContainerStarted","Data":"bfe37f5e7f9dff25ac88692d9b40fe1f4cae3e969bd6e17642647273bc1377d3"}
Sep 05 19:54:15.805460 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:15.805117    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:15.824349 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:15.824290    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb" podStartSLOduration=1.806660055 podStartE2EDuration="3.82427434s" totalImagesPullingTime="2.017614285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:15.822518639 +0000 UTC m=+665.239592062" watchObservedRunningTime="2026-09-05 19:54:15.82427434 +0000 UTC m=+665.241347762"
Sep 05 19:54:25.601387 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:25.601353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:25.661093 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:25.661056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:26.810576 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:26.810540    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"
Sep 05 19:54:31.664898 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:31.664859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:31.762274 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:31.762232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:32.106487 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:32.106453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:44.211590 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:44.211557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:44.545540 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:44.545451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:51.072452 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:51.072412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:51.174126 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:51.174091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:51.236380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:51.236345    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:51.329952 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:51.329858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:51.457630 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:51.457598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:55.110777 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:55.110739    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:54:55.207277 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:54:55.207247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:25.752817 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:25.752781    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:27.731874 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:27.731835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:27.833319 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:27.833283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.165058 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.164958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.257090 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.257055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.319993 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.319949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.537463 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.537380    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.757825 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.757791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.858932 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.858905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:29.998914 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:29.998877    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:55:39.103350 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:39.103285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:55:39.204449 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:55:39.204418    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:01.223062 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:01.223020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:01.734890 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:01.734853    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:56:14.059453 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:14.059392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:14.824066 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:14.824034    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:56:22.929850 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.929813    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"]
Sep 05 19:56:22.933301 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.933284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:22.936970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.936944    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnwjp\""
Sep 05 19:56:22.936970 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.936969    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 05 19:56:22.937234 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.937217    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 05 19:56:22.937289 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.937245    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 05 19:56:22.943824 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:22.943802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"]
Sep 05 19:56:23.021993 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.021952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:23.025507 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.025594 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e3c99b-b33e-4957-b278-dea511e3acdf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.025594 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.025677 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.025677 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9swl\" (UniqueName: \"kubernetes.io/projected/87e3c99b-b33e-4957-b278-dea511e3acdf-kube-api-access-j9swl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.025750 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.025683    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127092 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127092 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127354 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9swl\" (UniqueName: \"kubernetes.io/projected/87e3c99b-b33e-4957-b278-dea511e3acdf-kube-api-access-j9swl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127354 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127354 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127506 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e3c99b-b33e-4957-b278-dea511e3acdf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127506 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127798 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.127896 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.127820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.129492 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.129467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e3c99b-b33e-4957-b278-dea511e3acdf-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.129974 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.129950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e3c99b-b33e-4957-b278-dea511e3acdf-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.136094 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.136072    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9swl\" (UniqueName: \"kubernetes.io/projected/87e3c99b-b33e-4957-b278-dea511e3acdf-kube-api-access-j9swl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7\" (UID: \"87e3c99b-b33e-4957-b278-dea511e3acdf\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.243679 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.243593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:23.514511 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.514369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"]
Sep 05 19:56:23.517405 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:56:23.517376    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e3c99b_b33e_4957_b278_dea511e3acdf.slice/crio-a7e22d12322183759ff78e6a5bd192724be55ccc24d2d0462cdc4b2629b5ddb8 WatchSource:0}: Error finding container a7e22d12322183759ff78e6a5bd192724be55ccc24d2d0462cdc4b2629b5ddb8: Status 404 returned error can't find the container with id a7e22d12322183759ff78e6a5bd192724be55ccc24d2d0462cdc4b2629b5ddb8
Sep 05 19:56:23.518979 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.518956    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:56:23.931116 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:23.931075    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:56:24.389371 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:24.389322    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7" event={"ID":"87e3c99b-b33e-4957-b278-dea511e3acdf","Type":"ContainerStarted","Data":"a7e22d12322183759ff78e6a5bd192724be55ccc24d2d0462cdc4b2629b5ddb8"}
Sep 05 19:56:25.394977 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:25.394937    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7" event={"ID":"87e3c99b-b33e-4957-b278-dea511e3acdf","Type":"ContainerStarted","Data":"0f052964b9dabe036d27c887fa31c8095e3fb8aeae50d013c4c60736d9ae07b5"}
Sep 05 19:56:25.413239 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:25.413161    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7" podStartSLOduration=2.403365363 podStartE2EDuration="3.413144506s" totalImagesPullingTime="1.009779143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:56:25.412322681 +0000 UTC m=+794.829396104" watchObservedRunningTime="2026-09-05 19:56:25.413144506 +0000 UTC m=+794.830217928"
Sep 05 19:56:33.243716 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:33.243673    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:33.244291 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:33.243731    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:33.256574 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:33.256550    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:33.441871 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:33.441800    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7"
Sep 05 19:56:37.341292 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:37.341261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:38.231591 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:38.231552    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:56:47.851911 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:47.851877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:56:49.333638 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:56:49.333600    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:57:03.328386 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.328351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"]
Sep 05 19:57:03.332804 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.332783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.335361 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.335339    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 05 19:57:03.345695 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.345671    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"]
Sep 05 19:57:03.367369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.367346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:57:03.477366 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjg9\" (UniqueName: \"kubernetes.io/projected/4005545a-cb28-4e95-bdf3-6e2fd375d457-kube-api-access-2vjg9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.477521 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.477521 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477413    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.477521 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.477521 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.477697 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.477535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4005545a-cb28-4e95-bdf3-6e2fd375d457-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578328 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4005545a-cb28-4e95-bdf3-6e2fd375d457-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjg9\" (UniqueName: \"kubernetes.io/projected/4005545a-cb28-4e95-bdf3-6e2fd375d457-kube-api-access-2vjg9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578522 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578809 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578766    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.578876 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.578864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.580604 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.580584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4005545a-cb28-4e95-bdf3-6e2fd375d457-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.580890 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.580870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4005545a-cb28-4e95-bdf3-6e2fd375d457-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.586647 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.586627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjg9\" (UniqueName: \"kubernetes.io/projected/4005545a-cb28-4e95-bdf3-6e2fd375d457-kube-api-access-2vjg9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phr4h\" (UID: \"4005545a-cb28-4e95-bdf3-6e2fd375d457\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.643232 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.643215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:03.918733 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:03.918663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"]
Sep 05 19:57:03.921364 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:57:03.921336    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4005545a_cb28_4e95_bdf3_6e2fd375d457.slice/crio-a20b622c4d2d0f94a27272acc4b7048c2d5a4c758441544391c4fce5324687c5 WatchSource:0}: Error finding container a20b622c4d2d0f94a27272acc4b7048c2d5a4c758441544391c4fce5324687c5: Status 404 returned error can't find the container with id a20b622c4d2d0f94a27272acc4b7048c2d5a4c758441544391c4fce5324687c5
Sep 05 19:57:04.586682 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:04.586645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h" event={"ID":"4005545a-cb28-4e95-bdf3-6e2fd375d457","Type":"ContainerStarted","Data":"bc6ec5e2f637e226de2a383fb0612b5e60ff5bd951d3325debf0301dccbf6e71"}
Sep 05 19:57:04.587146 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:04.586722    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h" event={"ID":"4005545a-cb28-4e95-bdf3-6e2fd375d457","Type":"ContainerStarted","Data":"a20b622c4d2d0f94a27272acc4b7048c2d5a4c758441544391c4fce5324687c5"}
Sep 05 19:57:04.607500 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:04.607441    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h" podStartSLOduration=1.429268628 podStartE2EDuration="1.607423645s" totalImagesPullingTime="178.155017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:57:04.605947794 +0000 UTC m=+834.023021255" watchObservedRunningTime="2026-09-05 19:57:04.607423645 +0000 UTC m=+834.024497095"
Sep 05 19:57:05.132503 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:05.132465    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:57:13.644245 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:13.644178    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:13.644845 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:13.644285    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:13.656562 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:13.656533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:14.644837 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:14.644809    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phr4h"
Sep 05 19:57:26.321611 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:26.321572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:57:28.227132 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:28.227097    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:57:32.993717 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:32.993668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:57:33.093943 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:33.093902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:57:54.960657 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:54.960623    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:57:55.063348 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:57:55.063310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:10.990374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:10.990287    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:58:10.992879 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:10.992856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 19:58:18.227582 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:18.227539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:40.345673 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:40.345632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:40.433965 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:40.433928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:41.425664 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:41.425617    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:41.514605 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:41.514569    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:56.306658 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:56.306621    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:56.421584 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:56.421542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:56.699508 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:56.699474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:56.830333 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:56.830272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:57.689882 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:57.689845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:57.788692 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:57.788652    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:58.107336 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:58.107302    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:59.794028 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:59.793984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:58:59.892374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:58:59.892337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:00.531274 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:00.531240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:59:01.009060 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:01.009026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:01.107888 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:01.107848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:01.591999 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:01.591966    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:01.690344 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:01.690308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:01.780771 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:01.780740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:04.249198 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:04.249156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:04.349719 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:04.349686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:04.471218 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:04.471165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:04.753945 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:04.753908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:05.164046 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:05.164001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:05.625026 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:05.624994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:05.919147 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:05.919030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:06.202360 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:06.202256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:07.306761 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:07.306711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:07.402703 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:07.402664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:08.601690 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:08.601661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:08.702641 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:08.702607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:08.914554 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:08.914463    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:09.012385 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:09.012354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:09.120848 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:09.120805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:09.190623 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:09.190460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:10.171976 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.171939    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 19:59:10.175743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.175720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.178807 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.178783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1fd18b-serving-cert\""
Sep 05 19:59:10.178932 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.178832    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 05 19:59:10.179573 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.179553    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 19:59:10.179877 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.179858    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 05 19:59:10.179993 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.179941    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 19:59:10.186218 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.186137    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 19:59:10.206153 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.206111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:10.263427 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.263384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.263632 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.263443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.263632 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.263484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7gw\" (UniqueName: \"kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.263632 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.263515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.309776 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.309737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:10.365002 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.364958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.365268 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.365015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.365268 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.365051    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7gw\" (UniqueName: \"kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.365268 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.365075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.365268 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:59:10.365158    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-serving-cert: secret "maas-api-e2e-autores-1fd18b-serving-cert" not found
Sep 05 19:59:10.365490 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:59:10.365274    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls podName:9ba54a10-6277-464b-9006-7b2383bd7ca0 nodeName:}" failed. No retries permitted until 2026-09-05 19:59:10.865251927 +0000 UTC m=+960.282325345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls") pod "maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" (UID: "9ba54a10-6277-464b-9006-7b2383bd7ca0") : secret "maas-api-e2e-autores-1fd18b-serving-cert" not found
Sep 05 19:59:10.365832 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.365811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.367531 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.367505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.376122 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.376096    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7gw\" (UniqueName: \"kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.525796 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.525667    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:10.801423 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.801319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:10.868613 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.868568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.871161 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.871138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") pod \"maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:10.899217 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:10.899172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:11.091707 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:11.091675    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 19:59:11.099827 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:11.099799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:11.374078 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:11.373991    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 19:59:11.378110 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:59:11.378076    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba54a10_6277_464b_9006_7b2383bd7ca0.slice/crio-e3eaa6b0a61377606954adf1b3b96be8097298a7213bb6625c0ce20d33f5276f WatchSource:0}: Error finding container e3eaa6b0a61377606954adf1b3b96be8097298a7213bb6625c0ce20d33f5276f: Status 404 returned error can't find the container with id e3eaa6b0a61377606954adf1b3b96be8097298a7213bb6625c0ce20d33f5276f
Sep 05 19:59:12.181787 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:12.181737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" event={"ID":"9ba54a10-6277-464b-9006-7b2383bd7ca0","Type":"ContainerStarted","Data":"e3eaa6b0a61377606954adf1b3b96be8097298a7213bb6625c0ce20d33f5276f"}
Sep 05 19:59:14.192694 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:14.192660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" event={"ID":"9ba54a10-6277-464b-9006-7b2383bd7ca0","Type":"ContainerStarted","Data":"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"}
Sep 05 19:59:14.193101 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:14.192783    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:14.212869 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:14.212816    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" podStartSLOduration=2.069034115 podStartE2EDuration="4.212801825s" totalImagesPullingTime="2.14376771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:14.210923358 +0000 UTC m=+963.627996785" watchObservedRunningTime="2026-09-05 19:59:14.212801825 +0000 UTC m=+963.629875247"
Sep 05 19:59:14.530358 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:14.530271    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:59:15.909456 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:15.909419    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 19:59:15.912941 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:15.912922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:15.915600 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:15.915576    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c038f394-kserve-self-signed-certs\""
Sep 05 19:59:15.923888 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:15.923866    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 19:59:16.017276 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.017276 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.017547 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.017547 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75r59\" (UniqueName: \"kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.017547 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.017681 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.017572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118331 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118536 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118536 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118536 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118536 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118765 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75r59\" (UniqueName: \"kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118765 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118872 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.118872 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.118825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.120743 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.120720    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.121077 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.121058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.127256 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.127233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75r59\" (UniqueName: \"kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59\") pod \"e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") " pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.216723 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.216635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:16.222589 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.222565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:16.494466 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:16.494438    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 19:59:16.496361 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:59:16.496326    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf78d497f_fcfe_443d_8c8b_078106816ec5.slice/crio-cb314685b741fa04491f662ca84ea79c56eba5c297d1c4b4276d25a301b41d5f WatchSource:0}: Error finding container cb314685b741fa04491f662ca84ea79c56eba5c297d1c4b4276d25a301b41d5f: Status 404 returned error can't find the container with id cb314685b741fa04491f662ca84ea79c56eba5c297d1c4b4276d25a301b41d5f
Sep 05 19:59:17.191848 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.191813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:17.208596 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.208567    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" event={"ID":"f78d497f-fcfe-443d-8c8b-078106816ec5","Type":"ContainerStarted","Data":"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"}
Sep 05 19:59:17.208754 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.208605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" event={"ID":"f78d497f-fcfe-443d-8c8b-078106816ec5","Type":"ContainerStarted","Data":"cb314685b741fa04491f662ca84ea79c56eba5c297d1c4b4276d25a301b41d5f"}
Sep 05 19:59:17.227108 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.227054    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" podStartSLOduration=2.227038983 podStartE2EDuration="2.227038983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:17.225059671 +0000 UTC m=+966.642133094" watchObservedRunningTime="2026-09-05 19:59:17.227038983 +0000 UTC m=+966.644112405"
Sep 05 19:59:17.339667 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.339610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:17.436215 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.436160    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:17.537770 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:17.537668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:20.202905 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:20.202879    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 19:59:20.219688 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:20.219650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:21.872621 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:21.872570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:22.017493 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:22.017451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:24.678380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:24.678344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:24.996912 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:24.996831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:25.274500 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.274408    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 19:59:25.282619 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.282595    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.285380 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.285359    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1dc7b1-openshift-default-dockercfg-scwtx\""
Sep 05 19:59:25.291769 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.291741    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 19:59:25.308280 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.308243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:25.404966 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.404934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.404976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405432 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405432 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.405432 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.405341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66z4g\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.505818 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.505818 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506067 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.505999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66z4g\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506434 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.506316    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506434 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.506413    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506533 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.506479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506533 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.506513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.506908 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.506889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.508144 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.508125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.508742 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.508718    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.515675 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.515649    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.515786 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.515753    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66z4g\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g\") pod \"e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") " pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.594648 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.594607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:25.664225 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.664100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:25.875828 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:25.875742    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 19:59:25.879964 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:59:25.879938    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd514b6d1_890c_4102_9e24_f792dfb73ae6.slice/crio-4de59a4c449d41460a7d32759369dc2c490cac19a2732ec1349d8ba5db90566e WatchSource:0}: Error finding container 4de59a4c449d41460a7d32759369dc2c490cac19a2732ec1349d8ba5db90566e: Status 404 returned error can't find the container with id 4de59a4c449d41460a7d32759369dc2c490cac19a2732ec1349d8ba5db90566e
Sep 05 19:59:26.222871 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:26.222769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:26.222871 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:26.222834    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:26.235025 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:26.234997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:26.251472 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:26.251433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerStarted","Data":"4de59a4c449d41460a7d32759369dc2c490cac19a2732ec1349d8ba5db90566e"}
Sep 05 19:59:26.262964 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:26.262933    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 19:59:28.351800 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:28.351751    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:28.352163 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:28.351843    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:28.352163 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:28.351881    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:29.270938 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:29.270901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerStarted","Data":"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"}
Sep 05 19:59:29.292963 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:29.292907    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podStartSLOduration=1.823630992 podStartE2EDuration="4.292892386s" totalImagesPullingTime="2.469261394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:29.292357662 +0000 UTC m=+978.709431088" watchObservedRunningTime="2026-09-05 19:59:29.292892386 +0000 UTC m=+978.709965808"
Sep 05 19:59:29.565026 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:29.564991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:29.595345 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:29.595295    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:29.600339 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:29.600316    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:30.275501 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:30.275470    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:30.276395 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:30.276373    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:31.133478 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:31.133442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:31.229376 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:31.229340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:31.511718 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:31.511619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:31.869374 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:31.869338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:59:32.402059 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:32.402026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:32.500421 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:32.500385    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:32.928124 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:32.928086    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:33.025307 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:33.025269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:33.123551 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:33.123517    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:33.489424 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:33.489391    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:33.767594 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:33.767503    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:34.187712 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:34.187661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:34.940119 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:34.940078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:38.457074 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:38.457039    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:48.668267 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:48.668223    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:49.249558 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:49.249517    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:49.318764 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:49.318730    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:49.419626 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:49.419590    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:49.520178 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:49.520088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:49.591075 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:49.591043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:50.024026 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:50.023985    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:50.111918 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:50.111832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:50.213673 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:50.213636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:50.520844 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:50.520763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:50.859260 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:50.859230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:52.042077 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.042040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 19:59:52.382810 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.382778    2575 generic.go:356] "Generic (PLEG): container finished" podID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerID="d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1" exitCode=0
Sep 05 19:59:52.382977 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.382825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerDied","Data":"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"}
Sep 05 19:59:52.383208 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.383175    2575 scope.go:117] "RemoveContainer" containerID="d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"
Sep 05 19:59:52.384283 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.384248    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:52.384386 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.384337    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:52.384448 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:52.384388    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 19:59:53.392785 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:53.392739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerStarted","Data":"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"}
Sep 05 19:59:53.493840 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:53.493804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:53.595408 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:53.595369    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:53.595661 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:53.595630    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 05 19:59:53.595878 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:53.595842    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 05 19:59:54.595452 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:54.595413    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 05 19:59:54.596047 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:54.596014    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 05 19:59:55.595240 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:55.595198    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 05 19:59:55.595467 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:55.595278    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 05 19:59:55.888165 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:55.888068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:55.988212 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:55.985915    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 19:59:55.993131 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:55.993105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.003032 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.002995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e1fbf0a6-serving-cert\""
Sep 05 19:59:56.006776 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.006746    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 19:59:56.012445 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.011870    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:56.071823 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.071786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:56.112299 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.112264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.112489 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.112308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.112489 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.112337    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m26g9\" (UniqueName: \"kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.112579 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.112485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.213269 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.213144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.213269 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.213263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.213512 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.213281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.213512 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:59:56.213292    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-serving-cert: secret "maas-api-e2e-migrate-e1fbf0a6-serving-cert" not found
Sep 05 19:59:56.213512 ip-10-0-129-128 kubenswrapper[2575]: E0905 19:59:56.213393    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls podName:edc6bbca-7641-4db7-ba66-c60d354e9f9d nodeName:}" failed. No retries permitted until 2026-09-05 19:59:56.713372752 +0000 UTC m=+1006.130446157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls") pod "maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" (UID: "edc6bbca-7641-4db7-ba66-c60d354e9f9d") : secret "maas-api-e2e-migrate-e1fbf0a6-serving-cert" not found
Sep 05 19:59:56.213512 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.213297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m26g9\" (UniqueName: \"kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.214179 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.214157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.215953 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.215930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.221835 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.221815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m26g9\" (UniqueName: \"kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.526578 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.526495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:56.598393 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.598355    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 05 19:59:56.598594 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.598451    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 05 19:59:56.620733 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.620682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 19:59:56.719899 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.719844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.723212 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.723164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") pod \"maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:56.922730 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:56.922691    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:57.219494 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:57.219463    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 19:59:57.222315 ip-10-0-129-128 kubenswrapper[2575]: W0905 19:59:57.222283    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc6bbca_7641_4db7_ba66_c60d354e9f9d.slice/crio-dcb96377bbde5753107fc6b96108caf4a0677d802d9aabcbf0805d6077dc6e20 WatchSource:0}: Error finding container dcb96377bbde5753107fc6b96108caf4a0677d802d9aabcbf0805d6077dc6e20: Status 404 returned error can't find the container with id dcb96377bbde5753107fc6b96108caf4a0677d802d9aabcbf0805d6077dc6e20
Sep 05 19:59:57.416369 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:57.416331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" event={"ID":"edc6bbca-7641-4db7-ba66-c60d354e9f9d","Type":"ContainerStarted","Data":"dcb96377bbde5753107fc6b96108caf4a0677d802d9aabcbf0805d6077dc6e20"}
Sep 05 19:59:57.595299 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:57.595244    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 05 19:59:57.595419 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:57.595302    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 05 19:59:58.422930 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.422892    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" event={"ID":"edc6bbca-7641-4db7-ba66-c60d354e9f9d","Type":"ContainerStarted","Data":"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"}
Sep 05 19:59:58.423364 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.423003    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 19:59:58.441841 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.441787    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" podStartSLOduration=3.138270315 podStartE2EDuration="3.441769394s" totalImagesPullingTime="303.499079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:58.440348038 +0000 UTC m=+1007.857421462" watchObservedRunningTime="2026-09-05 19:59:58.441769394 +0000 UTC m=+1007.858842818"
Sep 05 19:59:58.875139 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.875102    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:58.875601 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.875557    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 19:59:58.877164 ip-10-0-129-128 kubenswrapper[2575]: I0905 19:59:58.877135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 20:00:00.140306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.140270    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:00.148778 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.148741    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"]
Sep 05 20:00:00.148947 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.148856    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"
Sep 05 20:00:00.151695 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.151537    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:00:00.153354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.153327    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:00.153491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.153442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"
Sep 05 20:00:00.163105 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.162870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"
Sep 05 20:00:00.168735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.168701    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"]
Sep 05 20:00:00.172440 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.172420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"
Sep 05 20:00:00.490971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.490936    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"]
Sep 05 20:00:00.492800 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:00.492777    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:00.492800 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:00:00.492793    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffeb245_877f_49a3_8e3d_43642f220b83.slice/crio-93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e WatchSource:0}: Error finding container 93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e: Status 404 returned error can't find the container with id 93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e
Sep 05 20:00:00.493629 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:00:00.493603    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84532062_d20e_47ea_b353_801238a0c15a.slice/crio-700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9 WatchSource:0}: Error finding container 700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9: Status 404 returned error can't find the container with id 700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9
Sep 05 20:00:01.438536 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:01.438482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt" event={"ID":"fffeb245-877f-49a3-8e3d-43642f220b83","Type":"ContainerStarted","Data":"93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e"}
Sep 05 20:00:01.439838 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:01.439810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerStarted","Data":"700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9"}
Sep 05 20:00:03.453388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:03.453346    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt" event={"ID":"fffeb245-877f-49a3-8e3d-43642f220b83","Type":"ContainerStarted","Data":"a4e76831c23dcc8c0835aa6f9b15820070b69658d13f3d5876d936ca6f2881d5"}
Sep 05 20:00:03.454880 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:03.454854    2575 generic.go:356] "Generic (PLEG): container finished" podID="84532062-d20e-47ea-b353-801238a0c15a" containerID="a32c6943c3481063214f454be1d4438e649adfce52088852d4e4a0d4e2d81092" exitCode=7
Sep 05 20:00:03.454880 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:03.454882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerDied","Data":"a32c6943c3481063214f454be1d4438e649adfce52088852d4e4a0d4e2d81092"}
Sep 05 20:00:03.455059 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:03.455046    2575 scope.go:117] "RemoveContainer" containerID="a32c6943c3481063214f454be1d4438e649adfce52088852d4e4a0d4e2d81092"
Sep 05 20:00:03.471870 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:03.471824    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt" podStartSLOduration=1.228018721 podStartE2EDuration="3.471803682s" totalImagesPullingTime="2.243784961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:03.469691009 +0000 UTC m=+1012.886764448" watchObservedRunningTime="2026-09-05 20:00:03.471803682 +0000 UTC m=+1012.888877106"
Sep 05 20:00:04.432459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.432383    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 20:00:04.462522 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.462479    2575 generic.go:356] "Generic (PLEG): container finished" podID="fffeb245-877f-49a3-8e3d-43642f220b83" containerID="a4e76831c23dcc8c0835aa6f9b15820070b69658d13f3d5876d936ca6f2881d5" exitCode=0
Sep 05 20:00:04.462959 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.462618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt" event={"ID":"fffeb245-877f-49a3-8e3d-43642f220b83","Type":"ContainerDied","Data":"a4e76831c23dcc8c0835aa6f9b15820070b69658d13f3d5876d936ca6f2881d5"}
Sep 05 20:00:04.464829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.464801    2575 generic.go:356] "Generic (PLEG): container finished" podID="84532062-d20e-47ea-b353-801238a0c15a" containerID="467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442" exitCode=7
Sep 05 20:00:04.464955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.464869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerDied","Data":"467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442"}
Sep 05 20:00:04.464955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.464912    2575 scope.go:117] "RemoveContainer" containerID="a32c6943c3481063214f454be1d4438e649adfce52088852d4e4a0d4e2d81092"
Sep 05 20:00:04.465097 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:04.465041    2575 scope.go:117] "RemoveContainer" containerID="467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442"
Sep 05 20:00:04.465373 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:00:04.465349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv_odh-ai-gateway-infra(84532062-d20e-47ea-b353-801238a0c15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" podUID="84532062-d20e-47ea-b353-801238a0c15a"
Sep 05 20:00:05.470955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:05.470930    2575 scope.go:117] "RemoveContainer" containerID="467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442"
Sep 05 20:00:05.471382 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:00:05.471171    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv_odh-ai-gateway-infra(84532062-d20e-47ea-b353-801238a0c15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" podUID="84532062-d20e-47ea-b353-801238a0c15a"
Sep 05 20:00:05.603425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:05.603402    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"
Sep 05 20:00:06.476534 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:06.476493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt" event={"ID":"fffeb245-877f-49a3-8e3d-43642f220b83","Type":"ContainerDied","Data":"93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e"}
Sep 05 20:00:06.476534 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:06.476541    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ec9a87a0d490a777494db68236d0ebea187af9daee15b49c9fc752fc80463e"
Sep 05 20:00:06.476985 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:06.476503    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"
Sep 05 20:00:08.334945 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:08.334885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:08.434215 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:08.434157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:08.555420 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:08.555390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:08.833032 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:08.832994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:00:14.129634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:14.129595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:15.324385 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:15.324350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:15.391246 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:15.391217    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:15.491439 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:15.491401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:15.655234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:15.655125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:18.037387 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:18.037358    2575 scope.go:117] "RemoveContainer" containerID="467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442"
Sep 05 20:00:18.542871 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:18.542839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerStarted","Data":"323bd216a32983b3259c2363c0500c23156d092eea743f2fa0ef27fcb558c56e"}
Sep 05 20:00:18.561649 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:18.561602    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" podStartSLOduration=16.322444709 podStartE2EDuration="18.561588679s" totalImagesPullingTime="2.23914397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:18.559586181 +0000 UTC m=+1027.976659605" watchObservedRunningTime="2026-09-05 20:00:18.561588679 +0000 UTC m=+1027.978662102"
Sep 05 20:00:19.549070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:19.549033    2575 generic.go:356] "Generic (PLEG): container finished" podID="84532062-d20e-47ea-b353-801238a0c15a" containerID="323bd216a32983b3259c2363c0500c23156d092eea743f2fa0ef27fcb558c56e" exitCode=0
Sep 05 20:00:19.549573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:19.549108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerDied","Data":"323bd216a32983b3259c2363c0500c23156d092eea743f2fa0ef27fcb558c56e"}
Sep 05 20:00:19.549573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:19.549170    2575 scope.go:117] "RemoveContainer" containerID="467a8a781043307c25fdd73cad792bf84fe6f91a68c83060a48ba0e3c4bad442"
Sep 05 20:00:19.571605 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:19.571577    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:20.689046 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:20.689018    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"
Sep 05 20:00:21.561043 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.561007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv" event={"ID":"84532062-d20e-47ea-b353-801238a0c15a","Type":"ContainerDied","Data":"700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9"}
Sep 05 20:00:21.561242 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.561061    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700b95001f16c6d66ae2a80e2f4de3db8b256965ad4bb530d31f2002e17f88c9"
Sep 05 20:00:21.561242 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.561019    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"
Sep 05 20:00:21.589443 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.589413    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:21.595669 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.595647    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e1fbf0a6-29810640-sc9jv"]
Sep 05 20:00:21.641790 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.641760    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:21.740431 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:21.740401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:23.043373 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:23.043342    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84532062-d20e-47ea-b353-801238a0c15a" path="/var/lib/kubelet/pods/84532062-d20e-47ea-b353-801238a0c15a/volumes"
Sep 05 20:00:27.337828 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.337795    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 20:00:27.338285 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.338026    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" podUID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" containerName="maas-api" containerID="cri-o://b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5" gracePeriod=30
Sep 05 20:00:27.360440 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.360411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:27.588846 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.588784    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 20:00:27.591759 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.591734    2575 generic.go:356] "Generic (PLEG): container finished" podID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" containerID="b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5" exitCode=0
Sep 05 20:00:27.591881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.591770    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"
Sep 05 20:00:27.591881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.591772    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" event={"ID":"edc6bbca-7641-4db7-ba66-c60d354e9f9d","Type":"ContainerDied","Data":"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"}
Sep 05 20:00:27.591881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.591836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8" event={"ID":"edc6bbca-7641-4db7-ba66-c60d354e9f9d","Type":"ContainerDied","Data":"dcb96377bbde5753107fc6b96108caf4a0677d802d9aabcbf0805d6077dc6e20"}
Sep 05 20:00:27.591881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.591872    2575 scope.go:117] "RemoveContainer" containerID="b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"
Sep 05 20:00:27.600983 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.600968    2575 scope.go:117] "RemoveContainer" containerID="b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"
Sep 05 20:00:27.601279 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:00:27.601261    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5\": container with ID starting with b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5 not found: ID does not exist" containerID="b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"
Sep 05 20:00:27.601324 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.601294    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5"} err="failed to get container status \"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5\": rpc error: code = NotFound desc = could not find container \"b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5\": container with ID starting with b4bbab4f8ff0dfc64e7ba67051b96e5b32583bd51130cb15eb238178696a51b5 not found: ID does not exist"
Sep 05 20:00:27.618023 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.617987    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") pod \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") "
Sep 05 20:00:27.618161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.618083    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs\") pod \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") "
Sep 05 20:00:27.618161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.618135    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca\") pod \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") "
Sep 05 20:00:27.618332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.618171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m26g9\" (UniqueName: \"kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9\") pod \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\" (UID: \"edc6bbca-7641-4db7-ba66-c60d354e9f9d\") "
Sep 05 20:00:27.618539 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.618513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edc6bbca-7641-4db7-ba66-c60d354e9f9d" (UID: "edc6bbca-7641-4db7-ba66-c60d354e9f9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:00:27.620288 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.620264    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "edc6bbca-7641-4db7-ba66-c60d354e9f9d" (UID: "edc6bbca-7641-4db7-ba66-c60d354e9f9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:00:27.620376 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.620302    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9" (OuterVolumeSpecName: "kube-api-access-m26g9") pod "edc6bbca-7641-4db7-ba66-c60d354e9f9d" (UID: "edc6bbca-7641-4db7-ba66-c60d354e9f9d"). InnerVolumeSpecName "kube-api-access-m26g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:00:27.620376 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.620343    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edc6bbca-7641-4db7-ba66-c60d354e9f9d" (UID: "edc6bbca-7641-4db7-ba66-c60d354e9f9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:00:27.719593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.719561    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:00:27.719593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.719586    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc6bbca-7641-4db7-ba66-c60d354e9f9d-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:00:27.719593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.719595    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6bbca-7641-4db7-ba66-c60d354e9f9d-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:00:27.719809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.719605    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m26g9\" (UniqueName: \"kubernetes.io/projected/edc6bbca-7641-4db7-ba66-c60d354e9f9d-kube-api-access-m26g9\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:00:27.915631 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.915596    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 20:00:27.919307 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:27.919280    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1fbf0a6-579bc8f7f9-vznc8"]
Sep 05 20:00:28.404262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404218    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:00:28.404680 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404656    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" containerName="maas-api"
Sep 05 20:00:28.404680 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404678    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" containerName="maas-api"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404705    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404713    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404728    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffeb245-877f-49a3-8e3d-43642f220b83" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404736    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffeb245-877f-49a3-8e3d-43642f220b83" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404744    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404751    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404763    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.404860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.404770    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.405263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.405082    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffeb245-877f-49a3-8e3d-43642f220b83" containerName="cleanup"
Sep 05 20:00:28.405263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.405102    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.405263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.405112    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:28.405263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.405129    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" containerName="maas-api"
Sep 05 20:00:28.409840 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.409820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.412686 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.412669    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a7428d13-openshift-default-dockercfg-5hz52\""
Sep 05 20:00:28.419791 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.419771    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:00:28.427649 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwl8l\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.427769 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.427769 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.427901 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.427901 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427814    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.427901 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.428074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.428074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.427999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.428179 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.428067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529109 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529109 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwl8l\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529857    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.529957 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.529936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.530029 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.530007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.531504 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.531485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.531829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.531811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.537836 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.537818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.538045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.538028    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwl8l\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l\") pod \"e2e-ait-a7428d13-openshift-default-bc64df448-74rcl\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") " pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.610257 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.610224    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:28.723374 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.723291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:28.757759 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.757726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:28.826230 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:28.826194    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:00:29.001722 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.001696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:00:29.007010 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:00:29.006980    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f228b1d_0896_4a3e_95d2_309f448b3a65.slice/crio-3418f9a61641e2c1f5f7a51411734c440b4cff65efee0f228b27a95104e565f7 WatchSource:0}: Error finding container 3418f9a61641e2c1f5f7a51411734c440b4cff65efee0f228b27a95104e565f7: Status 404 returned error can't find the container with id 3418f9a61641e2c1f5f7a51411734c440b4cff65efee0f228b27a95104e565f7
Sep 05 20:00:29.009532 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.009494    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:29.009652 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.009582    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:29.009652 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.009627    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:29.044207 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.044167    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc6bbca-7641-4db7-ba66-c60d354e9f9d" path="/var/lib/kubelet/pods/edc6bbca-7641-4db7-ba66-c60d354e9f9d/volumes"
Sep 05 20:00:29.604792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.604748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerStarted","Data":"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"}
Sep 05 20:00:29.605283 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.604830    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerStarted","Data":"3418f9a61641e2c1f5f7a51411734c440b4cff65efee0f228b27a95104e565f7"}
Sep 05 20:00:29.627276 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.627224    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podStartSLOduration=1.6272097840000002 podStartE2EDuration="1.627209784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:29.624888399 +0000 UTC m=+1039.041961819" watchObservedRunningTime="2026-09-05 20:00:29.627209784 +0000 UTC m=+1039.044283206"
Sep 05 20:00:29.724251 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.724181    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:29.845409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:29.845374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:30.729418 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:30.729385    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:31.012905 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:31.012817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:31.105314 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:31.105277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:31.613993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:31.613962    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:31.614908 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:31.614889    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:32.182829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.182797    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:00:32.183289 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.183273    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84532062-d20e-47ea-b353-801238a0c15a" containerName="cleanup"
Sep 05 20:00:32.186246 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.186227    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.190540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.190518    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a7428d13-serving-cert\""
Sep 05 20:00:32.205556 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.205533    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:00:32.229358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.229330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:32.263594 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.263563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.263746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.263611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.263746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.263690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kl8w\" (UniqueName: \"kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.263746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.263730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.321297 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.321252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:32.364825 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.364786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.365021 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.364847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.365021 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.364894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kl8w\" (UniqueName: \"kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.365021 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.364920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.365177 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:00:32.365032    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-serving-cert: secret "maas-api-e2e-ait-a7428d13-serving-cert" not found
Sep 05 20:00:32.365177 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:00:32.365114    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls podName:43c856e5-9089-456b-a7fe-76a7df77132a nodeName:}" failed. No retries permitted until 2026-09-05 20:00:32.86509089 +0000 UTC m=+1042.282164308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls") pod "maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" (UID: "43c856e5-9089-456b-a7fe-76a7df77132a") : secret "maas-api-e2e-ait-a7428d13-serving-cert" not found
Sep 05 20:00:32.365623 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.365602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.367324 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.367307    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.376998 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.376977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kl8w\" (UniqueName: \"kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.581563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.581527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:32.736640 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.736600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:32.835173 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.835079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:32.871081 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.871048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:32.873508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:32.873479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") pod \"maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:33.097376 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:33.097278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:33.371384 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:33.371321    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:00:33.373492 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:00:33.373465    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c856e5_9089_456b_a7fe_76a7df77132a.slice/crio-f8a7c255588c95b87700160c80a677bb4eef182338c412261a07027f19bdd873 WatchSource:0}: Error finding container f8a7c255588c95b87700160c80a677bb4eef182338c412261a07027f19bdd873: Status 404 returned error can't find the container with id f8a7c255588c95b87700160c80a677bb4eef182338c412261a07027f19bdd873
Sep 05 20:00:33.624818 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:33.624734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" event={"ID":"43c856e5-9089-456b-a7fe-76a7df77132a","Type":"ContainerStarted","Data":"f8a7c255588c95b87700160c80a677bb4eef182338c412261a07027f19bdd873"}
Sep 05 20:00:34.631415 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:34.631376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" event={"ID":"43c856e5-9089-456b-a7fe-76a7df77132a","Type":"ContainerStarted","Data":"2fc1bc72276de97d1443260e8f9f97b2d931a156c4267556ab5b351e82653b57"}
Sep 05 20:00:34.631899 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:34.631500    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:34.651614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:34.651561    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" podStartSLOduration=2.248000872 podStartE2EDuration="2.651545576s" totalImagesPullingTime="403.544704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:34.649480588 +0000 UTC m=+1044.066554012" watchObservedRunningTime="2026-09-05 20:00:34.651545576 +0000 UTC m=+1044.068619000"
Sep 05 20:00:40.641926 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:40.641897    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:00:47.616988 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.616842    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:00:47.616988 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.616915    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:00:47.720438 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.720404    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerID="5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551" exitCode=0
Sep 05 20:00:47.720601 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.720483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerDied","Data":"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"}
Sep 05 20:00:47.720920 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.720907    2575 scope.go:117] "RemoveContainer" containerID="5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"
Sep 05 20:00:47.722010 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.721984    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:47.722072 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.722048    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:47.722111 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:47.722082    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:00:48.727897 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:48.727861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerStarted","Data":"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"}
Sep 05 20:00:49.238710 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:49.238673    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:00:49.723736 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:49.723698    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:49.724043 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:49.724019    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:49.724091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:49.724071    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:50.724368 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:50.724327    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:50.724777 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:50.724384    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:51.201775 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:51.201739    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:51.724174 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:51.724136    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:51.724400 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:51.724226    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:52.724566 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:52.724531    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:52.724954 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:52.724594    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:53.154459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:53.154426    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:00:53.724242 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:53.724200    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:53.724427 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:53.724265    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:54.724476 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:54.724429    2575 patch_prober.go:28] interesting pod/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 05 20:00:54.724869 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:54.724503    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 05 20:00:55.728166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:55.728132    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:55.728614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:55.728548    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:00:55.729355 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:00:55.729335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:01:01.230358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:01.230323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:01.580603 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:01.580559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:02.681849 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.681818    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:01:02.682314 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.682038    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" podUID="43c856e5-9089-456b-a7fe-76a7df77132a" containerName="maas-api" containerID="cri-o://2fc1bc72276de97d1443260e8f9f97b2d931a156c4267556ab5b351e82653b57" gracePeriod=30
Sep 05 20:01:02.710701 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.710656    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:02.776906 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.776872    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:02.809841 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.809806    2575 generic.go:356] "Generic (PLEG): container finished" podID="43c856e5-9089-456b-a7fe-76a7df77132a" containerID="2fc1bc72276de97d1443260e8f9f97b2d931a156c4267556ab5b351e82653b57" exitCode=0
Sep 05 20:01:02.809991 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.809875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" event={"ID":"43c856e5-9089-456b-a7fe-76a7df77132a","Type":"ContainerDied","Data":"2fc1bc72276de97d1443260e8f9f97b2d931a156c4267556ab5b351e82653b57"}
Sep 05 20:01:02.877499 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.877461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:02.924547 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.924522    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:01:02.946199 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.946028    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:02.991892 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.991857    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs\") pod \"43c856e5-9089-456b-a7fe-76a7df77132a\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") "
Sep 05 20:01:02.992072 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.991927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") pod \"43c856e5-9089-456b-a7fe-76a7df77132a\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") "
Sep 05 20:01:02.992072 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.992043    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca\") pod \"43c856e5-9089-456b-a7fe-76a7df77132a\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") "
Sep 05 20:01:02.992166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.992140    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kl8w\" (UniqueName: \"kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w\") pod \"43c856e5-9089-456b-a7fe-76a7df77132a\" (UID: \"43c856e5-9089-456b-a7fe-76a7df77132a\") "
Sep 05 20:01:02.992375 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.992349    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43c856e5-9089-456b-a7fe-76a7df77132a" (UID: "43c856e5-9089-456b-a7fe-76a7df77132a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:02.992561 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.992543    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43c856e5-9089-456b-a7fe-76a7df77132a-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:02.994268 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.994246    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "43c856e5-9089-456b-a7fe-76a7df77132a" (UID: "43c856e5-9089-456b-a7fe-76a7df77132a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:02.994351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.994303    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43c856e5-9089-456b-a7fe-76a7df77132a" (UID: "43c856e5-9089-456b-a7fe-76a7df77132a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:02.994351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:02.994320    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w" (OuterVolumeSpecName: "kube-api-access-4kl8w") pod "43c856e5-9089-456b-a7fe-76a7df77132a" (UID: "43c856e5-9089-456b-a7fe-76a7df77132a"). InnerVolumeSpecName "kube-api-access-4kl8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:03.093648 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.093611    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:03.093648 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.093643    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43c856e5-9089-456b-a7fe-76a7df77132a-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:03.093848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.093658    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kl8w\" (UniqueName: \"kubernetes.io/projected/43c856e5-9089-456b-a7fe-76a7df77132a-kube-api-access-4kl8w\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:03.817074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.817040    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"
Sep 05 20:01:03.817074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.817067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4" event={"ID":"43c856e5-9089-456b-a7fe-76a7df77132a","Type":"ContainerDied","Data":"f8a7c255588c95b87700160c80a677bb4eef182338c412261a07027f19bdd873"}
Sep 05 20:01:03.817547 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.817148    2575 scope.go:117] "RemoveContainer" containerID="2fc1bc72276de97d1443260e8f9f97b2d931a156c4267556ab5b351e82653b57"
Sep 05 20:01:03.834154 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.834126    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:01:03.840140 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:03.840117    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a7428d13-6569bd6df9-8p8z4"]
Sep 05 20:01:04.003511 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.003479    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:01:04.003729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.003687    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy" containerID="cri-o://cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b" gracePeriod=30
Sep 05 20:01:04.029635 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.029594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:04.431433 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.431406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:04.615391 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.615350    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 20:01:04.631868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.631825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:04.704070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.703990    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:04.704442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.704423    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c856e5-9089-456b-a7fe-76a7df77132a" containerName="maas-api"
Sep 05 20:01:04.704442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.704441    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c856e5-9089-456b-a7fe-76a7df77132a" containerName="maas-api"
Sep 05 20:01:04.704660 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.704629    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c856e5-9089-456b-a7fe-76a7df77132a" containerName="maas-api"
Sep 05 20:01:04.707592 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.707569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.710284 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.710256    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c06406ce-openshift-default-dockercfg-wc7jx\""
Sep 05 20:01:04.719679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.719654    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:04.726240 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.726216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:04.809158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809370 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9dg\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809370 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809370 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809370 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.809644 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.809570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.910663 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9dg\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.910976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911584 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911584 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911584 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911584 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.911743 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.911598    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.914040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.914019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.914226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.914208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.919423 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.919400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:04.919535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:04.919515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9dg\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg\") pod \"e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") " pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:05.020462 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.020373    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:05.047637 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.047600    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c856e5-9089-456b-a7fe-76a7df77132a" path="/var/lib/kubelet/pods/43c856e5-9089-456b-a7fe-76a7df77132a/volumes"
Sep 05 20:01:05.050522 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.050485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:05.300092 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.300068    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:05.302174 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:01:05.302148    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52965cc0_e887_48fa_bda8_c8385aebfff5.slice/crio-eeabacf330a24b956bbdf95ec71b960850ddb09b2c85862d2f1d2a68c646a593 WatchSource:0}: Error finding container eeabacf330a24b956bbdf95ec71b960850ddb09b2c85862d2f1d2a68c646a593: Status 404 returned error can't find the container with id eeabacf330a24b956bbdf95ec71b960850ddb09b2c85862d2f1d2a68c646a593
Sep 05 20:01:05.304463 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.304429    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:05.304536 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.304494    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:05.304536 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.304521    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:05.721023 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.720990    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:05.831918 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.831866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerStarted","Data":"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"}
Sep 05 20:01:05.832180 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.831933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerStarted","Data":"eeabacf330a24b956bbdf95ec71b960850ddb09b2c85862d2f1d2a68c646a593"}
Sep 05 20:01:05.854730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:05.854679    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podStartSLOduration=1.854662333 podStartE2EDuration="1.854662333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:05.854136467 +0000 UTC m=+1075.271209894" watchObservedRunningTime="2026-09-05 20:01:05.854662333 +0000 UTC m=+1075.271735756"
Sep 05 20:01:06.021340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:06.021252    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:06.026023 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:06.025998    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:06.836874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:06.836840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:06.837856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:06.837835    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:07.161553 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:07.161467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:07.261241 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:07.261210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:07.880526 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:07.880490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:08.411728 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.411692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:08.508432 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.508400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:08.761725 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.761640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:08.885569 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.885541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:08.935625 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.935594    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:01:08.986262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:08.986229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:09.259490 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.259461    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:01:09.355794 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.355758    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.355996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.355800    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.355996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.355840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.355996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.355939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.355996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.355969    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwl8l\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.356255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356031    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.356255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356077    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.356255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356118    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.356255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356143    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token\") pod \"1f228b1d-0896-4a3e-95d2-309f448b3a65\" (UID: \"1f228b1d-0896-4a3e-95d2-309f448b3a65\") "
Sep 05 20:01:09.356460 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356257    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:09.356460 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356268    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:09.356460 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356349    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:09.356597 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356478    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:09.356597 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:09.356687 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356604    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.356687 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356625    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.356687 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356641    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.356687 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.356655    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.358212 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.358155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:01:09.358354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.358330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:09.358505 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.358483    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:09.358505 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.358493    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l" (OuterVolumeSpecName: "kube-api-access-hwl8l") pod "1f228b1d-0896-4a3e-95d2-309f448b3a65" (UID: "1f228b1d-0896-4a3e-95d2-309f448b3a65"). InnerVolumeSpecName "kube-api-access-hwl8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:09.457940 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.457905    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f228b1d-0896-4a3e-95d2-309f448b3a65-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.457940 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.457935    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.457940 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.457944    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.458378 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.457952    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f228b1d-0896-4a3e-95d2-309f448b3a65-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.458378 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.457961    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwl8l\" (UniqueName: \"kubernetes.io/projected/1f228b1d-0896-4a3e-95d2-309f448b3a65-kube-api-access-hwl8l\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:09.854883 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.854848    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerID="cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b" exitCode=0
Sep 05 20:01:09.855062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.854906    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"
Sep 05 20:01:09.855062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.854928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerDied","Data":"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"}
Sep 05 20:01:09.855062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.854982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl" event={"ID":"1f228b1d-0896-4a3e-95d2-309f448b3a65","Type":"ContainerDied","Data":"3418f9a61641e2c1f5f7a51411734c440b4cff65efee0f228b27a95104e565f7"}
Sep 05 20:01:09.855062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.855003    2575 scope.go:117] "RemoveContainer" containerID="cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"
Sep 05 20:01:09.864079 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.864061    2575 scope.go:117] "RemoveContainer" containerID="5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"
Sep 05 20:01:09.872159 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.872139    2575 scope.go:117] "RemoveContainer" containerID="cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"
Sep 05 20:01:09.872443 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:09.872423    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b\": container with ID starting with cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b not found: ID does not exist" containerID="cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"
Sep 05 20:01:09.872508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.872451    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b"} err="failed to get container status \"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b\": rpc error: code = NotFound desc = could not find container \"cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b\": container with ID starting with cbe780864f96cb8ee801d5fe48591383187c546b1a229626e3fe3c9e0e72ec6b not found: ID does not exist"
Sep 05 20:01:09.872508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.872467    2575 scope.go:117] "RemoveContainer" containerID="5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"
Sep 05 20:01:09.872703 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:09.872688    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551\": container with ID starting with 5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551 not found: ID does not exist" containerID="5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"
Sep 05 20:01:09.872743 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.872707    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551"} err="failed to get container status \"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551\": rpc error: code = NotFound desc = could not find container \"5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551\": container with ID starting with 5bdce3aa90f0d532b2f73e443ebef3f679672e51fe986770007840fca1e18551 not found: ID does not exist"
Sep 05 20:01:09.883307 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.883285    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:01:09.888063 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:09.888038    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a7428d13-openshift-default-bc64df448-74rcl"]
Sep 05 20:01:11.043743 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:11.043710    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" path="/var/lib/kubelet/pods/1f228b1d-0896-4a3e-95d2-309f448b3a65/volumes"
Sep 05 20:01:11.494168 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:11.494121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.242674 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.242642    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 20:01:12.243143 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.242887    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" podUID="9ba54a10-6277-464b-9006-7b2383bd7ca0" containerName="maas-api" containerID="cri-o://2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75" gracePeriod=30
Sep 05 20:01:12.272019 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.271986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.344498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.344461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.445366 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.445331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.508683 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.508653    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 20:01:12.558213 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.558157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.588715 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.588684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb7gw\" (UniqueName: \"kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw\") pod \"9ba54a10-6277-464b-9006-7b2383bd7ca0\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") "
Sep 05 20:01:12.588889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.588757    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca\") pod \"9ba54a10-6277-464b-9006-7b2383bd7ca0\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") "
Sep 05 20:01:12.588889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.588780    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") pod \"9ba54a10-6277-464b-9006-7b2383bd7ca0\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") "
Sep 05 20:01:12.588889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.588809    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs\") pod \"9ba54a10-6277-464b-9006-7b2383bd7ca0\" (UID: \"9ba54a10-6277-464b-9006-7b2383bd7ca0\") "
Sep 05 20:01:12.589221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.589181    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ba54a10-6277-464b-9006-7b2383bd7ca0" (UID: "9ba54a10-6277-464b-9006-7b2383bd7ca0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:12.591677 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.591623    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw" (OuterVolumeSpecName: "kube-api-access-wb7gw") pod "9ba54a10-6277-464b-9006-7b2383bd7ca0" (UID: "9ba54a10-6277-464b-9006-7b2383bd7ca0"). InnerVolumeSpecName "kube-api-access-wb7gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:12.591857 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.591836    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ba54a10-6277-464b-9006-7b2383bd7ca0" (UID: "9ba54a10-6277-464b-9006-7b2383bd7ca0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:12.591926 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.591856    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ba54a10-6277-464b-9006-7b2383bd7ca0" (UID: "9ba54a10-6277-464b-9006-7b2383bd7ca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:12.690255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.690218    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7gw\" (UniqueName: \"kubernetes.io/projected/9ba54a10-6277-464b-9006-7b2383bd7ca0-kube-api-access-wb7gw\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:12.690255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.690250    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba54a10-6277-464b-9006-7b2383bd7ca0-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:12.690255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.690263    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:12.690504 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.690272    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba54a10-6277-464b-9006-7b2383bd7ca0-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:12.816386 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.816357    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:12.870685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.870652    2575 generic.go:356] "Generic (PLEG): container finished" podID="9ba54a10-6277-464b-9006-7b2383bd7ca0" containerID="2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75" exitCode=0
Sep 05 20:01:12.870861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.870711    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"
Sep 05 20:01:12.870861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.870743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" event={"ID":"9ba54a10-6277-464b-9006-7b2383bd7ca0","Type":"ContainerDied","Data":"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"}
Sep 05 20:01:12.870861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.870796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg" event={"ID":"9ba54a10-6277-464b-9006-7b2383bd7ca0","Type":"ContainerDied","Data":"e3eaa6b0a61377606954adf1b3b96be8097298a7213bb6625c0ce20d33f5276f"}
Sep 05 20:01:12.870861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.870817    2575 scope.go:117] "RemoveContainer" containerID="2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"
Sep 05 20:01:12.879985 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.879965    2575 scope.go:117] "RemoveContainer" containerID="2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"
Sep 05 20:01:12.880251 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:12.880225    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75\": container with ID starting with 2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75 not found: ID does not exist" containerID="2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"
Sep 05 20:01:12.880316 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.880260    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75"} err="failed to get container status \"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75\": rpc error: code = NotFound desc = could not find container \"2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75\": container with ID starting with 2cc9ae84f116742b095dce8b91b2028b2173402d789ca30f1eb017c56fadeb75 not found: ID does not exist"
Sep 05 20:01:12.892732 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.892709    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 20:01:12.898738 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:12.898714    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1fd18b-79bd4bdcb6-8fhjg"]
Sep 05 20:01:13.042871 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:13.042835    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba54a10-6277-464b-9006-7b2383bd7ca0" path="/var/lib/kubelet/pods/9ba54a10-6277-464b-9006-7b2383bd7ca0/volumes"
Sep 05 20:01:17.894751 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:17.894715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:19.639084 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.639021    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" podUID="f78d497f-fcfe-443d-8c8b-078106816ec5" containerName="main" containerID="cri-o://fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932" gracePeriod=45
Sep 05 20:01:19.902543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.902477    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 20:01:19.905302 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.905271    2575 generic.go:356] "Generic (PLEG): container finished" podID="f78d497f-fcfe-443d-8c8b-078106816ec5" containerID="fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932" exitCode=0
Sep 05 20:01:19.905458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.905322    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"
Sep 05 20:01:19.905458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.905338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" event={"ID":"f78d497f-fcfe-443d-8c8b-078106816ec5","Type":"ContainerDied","Data":"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"}
Sep 05 20:01:19.905458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.905374    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw" event={"ID":"f78d497f-fcfe-443d-8c8b-078106816ec5","Type":"ContainerDied","Data":"cb314685b741fa04491f662ca84ea79c56eba5c297d1c4b4276d25a301b41d5f"}
Sep 05 20:01:19.905458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.905395    2575 scope.go:117] "RemoveContainer" containerID="fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"
Sep 05 20:01:19.914562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.914541    2575 scope.go:117] "RemoveContainer" containerID="fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"
Sep 05 20:01:19.914867 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:19.914840    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932\": container with ID starting with fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932 not found: ID does not exist" containerID="fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"
Sep 05 20:01:19.914952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:19.914881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932"} err="failed to get container status \"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932\": rpc error: code = NotFound desc = could not find container \"fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932\": container with ID starting with fd310e1cfb1b6168c01c7e0ddfcdf1f15d784a2b7968926235ad086715b99932 not found: ID does not exist"
Sep 05 20:01:20.057581 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057547    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75r59\" (UniqueName: \"kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.057816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057657    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.057816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057695    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.057816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057741    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.057816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057776    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.057816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.057819    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir\") pod \"f78d497f-fcfe-443d-8c8b-078106816ec5\" (UID: \"f78d497f-fcfe-443d-8c8b-078106816ec5\") "
Sep 05 20:01:20.058540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.058127    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home" (OuterVolumeSpecName: "home") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:20.058540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.058352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:20.058540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.058377    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:20.060718 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.060603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm" (OuterVolumeSpecName: "dshm") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:20.061084 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.061060    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:20.061179 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.061155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59" (OuterVolumeSpecName: "kube-api-access-75r59") pod "f78d497f-fcfe-443d-8c8b-078106816ec5" (UID: "f78d497f-fcfe-443d-8c8b-078106816ec5"). InnerVolumeSpecName "kube-api-access-75r59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159392    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-dshm\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159418    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-tmp-dir\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159429    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75r59\" (UniqueName: \"kubernetes.io/projected/f78d497f-fcfe-443d-8c8b-078106816ec5-kube-api-access-75r59\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159437    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78d497f-fcfe-443d-8c8b-078106816ec5-tls-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159444    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-home\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.159469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.159451    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78d497f-fcfe-443d-8c8b-078106816ec5-model-cache\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:20.227064 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.227036    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 20:01:20.231413 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:20.231390    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c038f394-kserve-85f78fc75f-4kkjw"]
Sep 05 20:01:21.043558 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:21.043515    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78d497f-fcfe-443d-8c8b-078106816ec5" path="/var/lib/kubelet/pods/f78d497f-fcfe-443d-8c8b-078106816ec5/volumes"
Sep 05 20:01:26.024010 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:26.023977    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:01:29.409234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.409177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:29.957656 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.957589    2575 generic.go:356] "Generic (PLEG): container finished" podID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerID="3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563" exitCode=0
Sep 05 20:01:29.957829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.957661    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerDied","Data":"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"}
Sep 05 20:01:29.958092 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.958068    2575 scope.go:117] "RemoveContainer" containerID="3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"
Sep 05 20:01:29.959132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.959103    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:29.959215 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.959170    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:29.959266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:29.959223    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:30.964759 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:30.964722    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerStarted","Data":"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"}
Sep 05 20:01:31.020787 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.020746    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:31.021002 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.020984    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:31.021055 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.021036    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:31.988405 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.988347    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"]
Sep 05 20:01:31.989111 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989086    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:31.989261 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989111    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:31.989261 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989151    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78d497f-fcfe-443d-8c8b-078106816ec5" containerName="main"
Sep 05 20:01:31.989261 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989160    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78d497f-fcfe-443d-8c8b-078106816ec5" containerName="main"
Sep 05 20:01:31.989261 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989178    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba54a10-6277-464b-9006-7b2383bd7ca0" containerName="maas-api"
Sep 05 20:01:31.989261 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989207    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba54a10-6277-464b-9006-7b2383bd7ca0" containerName="maas-api"
Sep 05 20:01:31.989542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989327    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba54a10-6277-464b-9006-7b2383bd7ca0" containerName="maas-api"
Sep 05 20:01:31.989542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989339    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:31.989542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989362    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:31.989542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.989372    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78d497f-fcfe-443d-8c8b-078106816ec5" containerName="main"
Sep 05 20:01:31.992102 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.992078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:31.994490 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.994463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c06406ce-rpsss\" (UID: \"134bb41c-e865-4651-8fa4-43d2e9969639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:31.996721 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.996698    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:01:31.997617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:31.997595    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:01:32.000941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.000921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"]
Sep 05 20:01:32.020760 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.020732    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:32.020902 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.020782    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:32.096605 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.096567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c06406ce-rpsss\" (UID: \"134bb41c-e865-4651-8fa4-43d2e9969639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:32.097670 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.097642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c06406ce-rpsss\" (UID: \"134bb41c-e865-4651-8fa4-43d2e9969639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:32.306617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.306535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:32.597862 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.597831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"]
Sep 05 20:01:32.979119 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.979075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss" event={"ID":"134bb41c-e865-4651-8fa4-43d2e9969639","Type":"ContainerStarted","Data":"457cc5efc727bd8510d7aff4797250f36a160824e1e99968d1ea2f1f8d0002ea"}
Sep 05 20:01:32.979319 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:32.979144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss" event={"ID":"134bb41c-e865-4651-8fa4-43d2e9969639","Type":"ContainerStarted","Data":"a925f71fe94cfea2ff080d94acbaf5341fc4710ce9d5352f06e01ef6bebddabc"}
Sep 05 20:01:33.000621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:33.000567    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss" podStartSLOduration=2.000554406 podStartE2EDuration="2.000554406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:32.99957811 +0000 UTC m=+1102.416651534" watchObservedRunningTime="2026-09-05 20:01:33.000554406 +0000 UTC m=+1102.417627829"
Sep 05 20:01:33.021254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:33.021227    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:33.021361 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:33.021270    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:33.993598 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:33.993494    2575 generic.go:356] "Generic (PLEG): container finished" podID="134bb41c-e865-4651-8fa4-43d2e9969639" containerID="457cc5efc727bd8510d7aff4797250f36a160824e1e99968d1ea2f1f8d0002ea" exitCode=0
Sep 05 20:01:33.993775 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:33.993608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss" event={"ID":"134bb41c-e865-4651-8fa4-43d2e9969639","Type":"ContainerDied","Data":"457cc5efc727bd8510d7aff4797250f36a160824e1e99968d1ea2f1f8d0002ea"}
Sep 05 20:01:34.021266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:34.021235    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:34.021658 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:34.021285    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:35.021213 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.021124    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:35.021409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.021214    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:35.159283 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.159258    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:35.224040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.223997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca\") pod \"134bb41c-e865-4651-8fa4-43d2e9969639\" (UID: \"134bb41c-e865-4651-8fa4-43d2e9969639\") "
Sep 05 20:01:35.224724 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.224676    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "134bb41c-e865-4651-8fa4-43d2e9969639" (UID: "134bb41c-e865-4651-8fa4-43d2e9969639"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:35.325306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:35.325273    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/134bb41c-e865-4651-8fa4-43d2e9969639-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:36.007382 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.007351    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"
Sep 05 20:01:36.007382 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.007376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss" event={"ID":"134bb41c-e865-4651-8fa4-43d2e9969639","Type":"ContainerDied","Data":"a925f71fe94cfea2ff080d94acbaf5341fc4710ce9d5352f06e01ef6bebddabc"}
Sep 05 20:01:36.007604 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.007427    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a925f71fe94cfea2ff080d94acbaf5341fc4710ce9d5352f06e01ef6bebddabc"
Sep 05 20:01:36.021268 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.021241    2575 patch_prober.go:28] interesting pod/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 05 20:01:36.021399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.021287    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 05 20:01:36.836740 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:36.836707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:37.025394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:37.025363    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:37.026239 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:37.026219    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:38.832138 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:38.832103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:38.926036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:38.926004    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:38.988041 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:38.988011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:39.042644 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.042607    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"]
Sep 05 20:01:39.042644 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.042647    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c06406ce-rpsss"]
Sep 05 20:01:39.051200 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.051150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:39.163149 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.163067    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:39.163342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.163308    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy" containerID="cri-o://161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373" gracePeriod=30
Sep 05 20:01:39.199035 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.198996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:39.794099 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.794059    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:39.893358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.893326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:39.993087 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:39.993050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:41.043404 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:41.043369    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="134bb41c-e865-4651-8fa4-43d2e9969639" path="/var/lib/kubelet/pods/134bb41c-e865-4651-8fa4-43d2e9969639/volumes"
Sep 05 20:01:41.292338 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:41.292300    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:42.530372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:42.530330    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:01:44.410244 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.410219    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:44.505802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.505698    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.505802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.505747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.505802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.505776    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.505919    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.505983    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506017    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data" (OuterVolumeSpecName: "istio-data") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:44.506071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506022    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:44.506321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506222    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:44.506321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll9dg\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg\") pod \"52965cc0-e887-48fa-bda8-c8385aebfff5\" (UID: \"52965cc0-e887-48fa-bda8-c8385aebfff5\") "
Sep 05 20:01:44.506585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506332    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:44.506585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506439    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:44.506684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506608    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.506684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506631    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52965cc0-e887-48fa-bda8-c8385aebfff5-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.506684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506644    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.506684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506657    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.506684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.506669    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.508060 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.508036    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:01:44.508358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.508335    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:44.508427 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.508360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token" (OuterVolumeSpecName: "istio-token") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:44.508427 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.508389    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg" (OuterVolumeSpecName: "kube-api-access-ll9dg") pod "52965cc0-e887-48fa-bda8-c8385aebfff5" (UID: "52965cc0-e887-48fa-bda8-c8385aebfff5"). InnerVolumeSpecName "kube-api-access-ll9dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:44.607351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.607320    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9dg\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-kube-api-access-ll9dg\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.607351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.607347    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.607351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.607356    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:44.607571 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:44.607366    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52965cc0-e887-48fa-bda8-c8385aebfff5-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:01:45.056409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.056380    2575 generic.go:356] "Generic (PLEG): container finished" podID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerID="161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373" exitCode=0
Sep 05 20:01:45.056525 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.056425    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"
Sep 05 20:01:45.056525 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.056438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerDied","Data":"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"}
Sep 05 20:01:45.056525 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.056477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf" event={"ID":"52965cc0-e887-48fa-bda8-c8385aebfff5","Type":"ContainerDied","Data":"eeabacf330a24b956bbdf95ec71b960850ddb09b2c85862d2f1d2a68c646a593"}
Sep 05 20:01:45.056525 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.056499    2575 scope.go:117] "RemoveContainer" containerID="161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"
Sep 05 20:01:45.064986 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.064969    2575 scope.go:117] "RemoveContainer" containerID="3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"
Sep 05 20:01:45.072406 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.072388    2575 scope.go:117] "RemoveContainer" containerID="161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"
Sep 05 20:01:45.072669 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:45.072647    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373\": container with ID starting with 161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373 not found: ID does not exist" containerID="161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"
Sep 05 20:01:45.072746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.072681    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373"} err="failed to get container status \"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373\": rpc error: code = NotFound desc = could not find container \"161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373\": container with ID starting with 161e8779f57ef7ab502df334ff1d67b8ca2c02253a175cf03043ad06c7b36373 not found: ID does not exist"
Sep 05 20:01:45.072746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.072704    2575 scope.go:117] "RemoveContainer" containerID="3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"
Sep 05 20:01:45.072978 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:01:45.072958    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563\": container with ID starting with 3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563 not found: ID does not exist" containerID="3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"
Sep 05 20:01:45.073017 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.072985    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563"} err="failed to get container status \"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563\": rpc error: code = NotFound desc = could not find container \"3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563\": container with ID starting with 3574c7836846aed4a237f6cdc077636dd0878e28db1ee929f5861cb80ea7f563 not found: ID does not exist"
Sep 05 20:01:45.078730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.078705    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:45.082688 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:45.082666    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c06406ce-openshift-default-79864468f6-2f2rf"]
Sep 05 20:01:47.043394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.043362    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" path="/var/lib/kubelet/pods/52965cc0-e887-48fa-bda8-c8385aebfff5/volumes"
Sep 05 20:01:47.165521 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.165485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:47.252331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.252297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:47.325155 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.325125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:47.422081 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.422051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:47.523736 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.523705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:47.834518 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:47.834477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:54.858082 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:54.858046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:55.095623 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.095590    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:55.522172 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.522131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:55.797385 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797305    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:01:55.797704 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797692    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:01:55.797746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797706    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:01:55.797746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797721    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:55.797746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797730    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f228b1d-0896-4a3e-95d2-309f448b3a65" containerName="istio-proxy"
Sep 05 20:01:55.797874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797746    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="134bb41c-e865-4651-8fa4-43d2e9969639" containerName="revoke-keys"
Sep 05 20:01:55.797874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797753    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="134bb41c-e865-4651-8fa4-43d2e9969639" containerName="revoke-keys"
Sep 05 20:01:55.797874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797815    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:01:55.797874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797841    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="134bb41c-e865-4651-8fa4-43d2e9969639" containerName="revoke-keys"
Sep 05 20:01:55.797874 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.797849    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:01:55.801113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.801095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.805755 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.805727    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-49af4f-openshift-default-dockercfg-ghmms\""
Sep 05 20:01:55.821348 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.821321    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:01:55.830003 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.829970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:55.909070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.909567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.909480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkpwm\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:55.921384 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:55.921349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:56.010479 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.010697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkpwm\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.010990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.011023    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011289 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.011221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011289 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.011280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.011493 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.011472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.013509 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.013488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.013619 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.013604    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.019266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.019237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.019432 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.019418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkpwm\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm\") pod \"e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") " pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.111904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.111864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:56.391903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.391875    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:01:56.393688 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:01:56.393664    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca950656_522c_4b8f_a2da_b45f21762f4a.slice/crio-cd24eccbe479550ad17cd7e61ab7b057b14724780bdb80583686d3f6def5c11f WatchSource:0}: Error finding container cd24eccbe479550ad17cd7e61ab7b057b14724780bdb80583686d3f6def5c11f: Status 404 returned error can't find the container with id cd24eccbe479550ad17cd7e61ab7b057b14724780bdb80583686d3f6def5c11f
Sep 05 20:01:56.395884 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.395850    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:56.395993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.395933    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:56.395993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:56.395974    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:01:57.124013 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:57.123975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerStarted","Data":"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"}
Sep 05 20:01:57.124487 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:57.124027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerStarted","Data":"cd24eccbe479550ad17cd7e61ab7b057b14724780bdb80583686d3f6def5c11f"}
Sep 05 20:01:57.147335 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:57.147285    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podStartSLOduration=2.147266707 podStartE2EDuration="2.147266707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:57.146408895 +0000 UTC m=+1126.563482319" watchObservedRunningTime="2026-09-05 20:01:57.147266707 +0000 UTC m=+1126.564340130"
Sep 05 20:01:58.112999 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:58.112959    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:58.118296 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:58.118270    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:58.129379 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:58.129355    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:58.129986 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:58.129968    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:01:59.005098 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:59.005059    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:59.104341 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:59.104304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:01:59.729394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:01:59.729358    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:02:00.253280 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.253248    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:00.253832 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.253816    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:02:00.253915 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.253834    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52965cc0-e887-48fa-bda8-c8385aebfff5" containerName="istio-proxy"
Sep 05 20:02:00.257341 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.257317    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.260287 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.260258    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:02:00.260287 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.260282    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 05 20:02:00.260533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.260514    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-49af4f-serving-cert\""
Sep 05 20:02:00.261253 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.261237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 20:02:00.261388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.261260    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 05 20:02:00.267709 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.267663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:00.290350 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.290314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:00.349743 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.349706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:00.353868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.353834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.354036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.353894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plffg\" (UniqueName: \"kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.354036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.353936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.354036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.353998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.455194 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.455159    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.455398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.455311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.455398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.455380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plffg\" (UniqueName: \"kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.455575 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.455425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.456011 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.455988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.456435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.456414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:00.458256 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.458228    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.458337 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.458233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.464750 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.464728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plffg\" (UniqueName: \"kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg\") pod \"maas-api-e2e-explicit-49af4f-54f5796b9-fqv52\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.574227 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.574201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:00.807807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.807775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:00.850806 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.850646    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:00.853140 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:02:00.853106    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351e516c_c3fd_4d5d_9bf8_eb017d5285ed.slice/crio-a87a51e16fde23c6bc78e01b2afcdbae9bc3648a097aa5c57666a36f529aaac9 WatchSource:0}: Error finding container a87a51e16fde23c6bc78e01b2afcdbae9bc3648a097aa5c57666a36f529aaac9: Status 404 returned error can't find the container with id a87a51e16fde23c6bc78e01b2afcdbae9bc3648a097aa5c57666a36f529aaac9
Sep 05 20:02:00.854720 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.854699    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 20:02:00.909039 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:00.909006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:01.145432 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:01.145347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" event={"ID":"351e516c-c3fd-4d5d-9bf8-eb017d5285ed","Type":"ContainerStarted","Data":"a87a51e16fde23c6bc78e01b2afcdbae9bc3648a097aa5c57666a36f529aaac9"}
Sep 05 20:02:02.151610 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:02.151570    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" event={"ID":"351e516c-c3fd-4d5d-9bf8-eb017d5285ed","Type":"ContainerStarted","Data":"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"}
Sep 05 20:02:02.152095 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:02.151637    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:02.173546 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:02.173487    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" podStartSLOduration=1.758633497 podStartE2EDuration="2.173472868s" totalImagesPullingTime="414.839371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:02:02.172532985 +0000 UTC m=+1131.589606409" watchObservedRunningTime="2026-09-05 20:02:02.173472868 +0000 UTC m=+1131.590546291"
Sep 05 20:02:08.162596 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:08.162567    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:13.708453 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:13.708410    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:16.425705 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:16.425670    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:02:18.239651 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.239564    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerID="7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8" exitCode=0
Sep 05 20:02:18.239651 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.239615    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerDied","Data":"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"}
Sep 05 20:02:18.240081 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.239986    2575 scope.go:117] "RemoveContainer" containerID="7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"
Sep 05 20:02:18.241023 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.240995    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:02:18.241080 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.241066    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:02:18.241121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:18.241105    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:02:19.246374 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:19.246334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerStarted","Data":"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"}
Sep 05 20:02:20.112388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.112349    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:02:20.112590 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.112570    2575 patch_prober.go:28] interesting pod/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 05 20:02:20.112641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.112625    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 05 20:02:20.137433 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.137400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:20.247807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.247771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:20.340649 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:20.340620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:21.113132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:21.113089    2575 patch_prober.go:28] interesting pod/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 05 20:02:21.113383 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:21.113156    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 05 20:02:22.113300 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:22.113250    2575 patch_prober.go:28] interesting pod/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 05 20:02:22.113701 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:22.113318    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 05 20:02:23.112800 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:23.112763    2575 patch_prober.go:28] interesting pod/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 05 20:02:23.112978 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:23.112816    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 05 20:02:23.915514 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:23.915477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:24.113047 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:24.113011    2575 patch_prober.go:28] interesting pod/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 05 20:02:24.113235 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:24.113077    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 05 20:02:25.116353 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:25.116316    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:02:25.116720 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:25.116576    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:02:25.117334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:25.117316    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:02:31.442145 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:31.442096    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:02:36.108658 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:36.108568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:36.262799 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:36.262765    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:41.837740 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:41.837700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:41.948475 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:41.948436    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:42.037975 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:42.037941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:42.533329 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:42.533295    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:43.043808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:43.043774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:02:52.019473 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:52.019437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:52.504889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:52.504846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.041933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.041892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.055708 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.055678    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:53.055949 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.055922    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerName="maas-api" containerID="cri-o://5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f" gracePeriod=30
Sep 05 20:02:53.130728 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.130690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.137297 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.137266    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"]
Sep 05 20:02:53.140157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.140134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:53.143331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.143303    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:02:53.151792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.151765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"]
Sep 05 20:02:53.158099 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.158066    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.44:8443/health\": dial tcp 10.133.0.44:8443: connect: connection refused"
Sep 05 20:02:53.191113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.191081    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.285413 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.285381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-588081-pl7zq\" (UID: \"3a085b51-c3b3-47ca-b304-8c51a7ebc126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:53.292928 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.292855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.298777 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.298756    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:53.356900 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.356870    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.386719 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.386689    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs\") pod \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") "
Sep 05 20:02:53.386896 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.386755    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca\") pod \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") "
Sep 05 20:02:53.386896 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.386881    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls\") pod \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") "
Sep 05 20:02:53.387027 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.386901    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plffg\" (UniqueName: \"kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg\") pod \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\" (UID: \"351e516c-c3fd-4d5d-9bf8-eb017d5285ed\") "
Sep 05 20:02:53.387122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.387089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "351e516c-c3fd-4d5d-9bf8-eb017d5285ed" (UID: "351e516c-c3fd-4d5d-9bf8-eb017d5285ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:02:53.387174 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.387126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-588081-pl7zq\" (UID: \"3a085b51-c3b3-47ca-b304-8c51a7ebc126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:53.387284 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.387267    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.387951 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.387927    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-588081-pl7zq\" (UID: \"3a085b51-c3b3-47ca-b304-8c51a7ebc126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:53.389048 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.389018    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "351e516c-c3fd-4d5d-9bf8-eb017d5285ed" (UID: "351e516c-c3fd-4d5d-9bf8-eb017d5285ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:02:53.389048 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.389031    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "351e516c-c3fd-4d5d-9bf8-eb017d5285ed" (UID: "351e516c-c3fd-4d5d-9bf8-eb017d5285ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:02:53.389234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.389036    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg" (OuterVolumeSpecName: "kube-api-access-plffg") pod "351e516c-c3fd-4d5d-9bf8-eb017d5285ed" (UID: "351e516c-c3fd-4d5d-9bf8-eb017d5285ed"). InnerVolumeSpecName "kube-api-access-plffg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:02:53.414351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.414321    2575 generic.go:356] "Generic (PLEG): container finished" podID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerID="5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f" exitCode=0
Sep 05 20:02:53.414484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.414370    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" event={"ID":"351e516c-c3fd-4d5d-9bf8-eb017d5285ed","Type":"ContainerDied","Data":"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"}
Sep 05 20:02:53.414484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.414404    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"
Sep 05 20:02:53.414484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.414438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52" event={"ID":"351e516c-c3fd-4d5d-9bf8-eb017d5285ed","Type":"ContainerDied","Data":"a87a51e16fde23c6bc78e01b2afcdbae9bc3648a097aa5c57666a36f529aaac9"}
Sep 05 20:02:53.414484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.414464    2575 scope.go:117] "RemoveContainer" containerID="5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"
Sep 05 20:02:53.422470 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.422439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.424675 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.424656    2575 scope.go:117] "RemoveContainer" containerID="5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"
Sep 05 20:02:53.425384 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:02:53.425222    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f\": container with ID starting with 5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f not found: ID does not exist" containerID="5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"
Sep 05 20:02:53.425384 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.425281    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f"} err="failed to get container status \"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f\": rpc error: code = NotFound desc = could not find container \"5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f\": container with ID starting with 5f910f208ba56e57907e2449dd388240158702bcad51c49531d7ff3c9069a42f not found: ID does not exist"
Sep 05 20:02:53.440436 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.440413    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:53.446546 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.446520    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-49af4f-54f5796b9-fqv52"]
Sep 05 20:02:53.454832 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.454812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:53.488864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.488826    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.488864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.488862    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.489061 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.488878    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plffg\" (UniqueName: \"kubernetes.io/projected/351e516c-c3fd-4d5d-9bf8-eb017d5285ed-kube-api-access-plffg\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.641885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.641851    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:53.729979 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:53.729937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"]
Sep 05 20:02:53.732197 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:02:53.732155    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a085b51_c3b3_47ca_b304_8c51a7ebc126.slice/crio-62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff WatchSource:0}: Error finding container 62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff: Status 404 returned error can't find the container with id 62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff
Sep 05 20:02:54.422077 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:54.422045    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a085b51-c3b3-47ca-b304-8c51a7ebc126" containerID="8c53e63acbdd531ec054715df784acd84634ef659e2a936af16dad4d189a6a96" exitCode=0
Sep 05 20:02:54.422478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:54.422124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq" event={"ID":"3a085b51-c3b3-47ca-b304-8c51a7ebc126","Type":"ContainerDied","Data":"8c53e63acbdd531ec054715df784acd84634ef659e2a936af16dad4d189a6a96"}
Sep 05 20:02:54.422478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:54.422172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq" event={"ID":"3a085b51-c3b3-47ca-b304-8c51a7ebc126","Type":"ContainerStarted","Data":"62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff"}
Sep 05 20:02:55.042989 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.042955    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" path="/var/lib/kubelet/pods/351e516c-c3fd-4d5d-9bf8-eb017d5285ed/volumes"
Sep 05 20:02:55.227708 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.227675    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:02:55.561073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.561047    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:55.711252 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.711141    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca\") pod \"3a085b51-c3b3-47ca-b304-8c51a7ebc126\" (UID: \"3a085b51-c3b3-47ca-b304-8c51a7ebc126\") "
Sep 05 20:02:55.711551 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.711524    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a085b51-c3b3-47ca-b304-8c51a7ebc126" (UID: "3a085b51-c3b3-47ca-b304-8c51a7ebc126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:02:55.812703 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:55.812667    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a085b51-c3b3-47ca-b304-8c51a7ebc126-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:02:56.432730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:56.432701    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"
Sep 05 20:02:56.432730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:56.432715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq" event={"ID":"3a085b51-c3b3-47ca-b304-8c51a7ebc126","Type":"ContainerDied","Data":"62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff"}
Sep 05 20:02:56.432959 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:56.432770    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ef26b23febf11d9563379d3d578122a8e8bc7e17e0f12b88cebebe315a5dff"
Sep 05 20:02:58.716445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:58.716404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:59.316161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:59.316108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:59.378302 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:59.378246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:59.453503 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:59.453458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:59.547115 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:59.547078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:02:59.647379 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:02:59.647294    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:00.215577 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.215545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:00.317099 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.317070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:00.417741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.417687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:00.551073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.550989    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:03:00.551256 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.551222    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy" containerID="cri-o://88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b" gracePeriod=30
Sep 05 20:03:00.578980 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:00.578947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:01.005034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.004998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:01.262715 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.262631    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:03:01.263341 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263313    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerName="maas-api"
Sep 05 20:03:01.263341 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263340    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerName="maas-api"
Sep 05 20:03:01.263544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263355    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a085b51-c3b3-47ca-b304-8c51a7ebc126" containerName="revoke-keys"
Sep 05 20:03:01.263544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263361    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a085b51-c3b3-47ca-b304-8c51a7ebc126" containerName="revoke-keys"
Sep 05 20:03:01.263544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263438    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a085b51-c3b3-47ca-b304-8c51a7ebc126" containerName="revoke-keys"
Sep 05 20:03:01.263544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.263452    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="351e516c-c3fd-4d5d-9bf8-eb017d5285ed" containerName="maas-api"
Sep 05 20:03:01.267594 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.267574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.271223 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.271174    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-0741a0-openshift-default-dockercfg-bz6tb\""
Sep 05 20:03:01.280589 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.280566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:03:01.292266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.292242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:01.362521 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94xvl\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362521 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362742    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362981 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362778    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362981 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.362981 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.362847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94xvl\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463505    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.463909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.464075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.463935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.464075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.464045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.464168 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.464149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.464415 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.464392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.466037 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.466012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.466167 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.466074    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.472690 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.472668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.472894 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.472872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94xvl\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl\") pod \"e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") " pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.516768 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.516650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:01.582829 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.582789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:01.822661 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.822624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:01.863777 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.863750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:03:01.866030 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:03:01.866003    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d95f0c7_3f34_4d53_9449_36cfc6ee2f55.slice/crio-b9ecefc4f9971d36d166256a092663e60db711e7c887a2cac598d83aa31967ee WatchSource:0}: Error finding container b9ecefc4f9971d36d166256a092663e60db711e7c887a2cac598d83aa31967ee: Status 404 returned error can't find the container with id b9ecefc4f9971d36d166256a092663e60db711e7c887a2cac598d83aa31967ee
Sep 05 20:03:01.868249 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.868214    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:01.868354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.868281    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:01.868354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:01.868316    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:02.466060 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.466024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" event={"ID":"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55","Type":"ContainerStarted","Data":"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"}
Sep 05 20:03:02.466555 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.466079    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" event={"ID":"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55","Type":"ContainerStarted","Data":"b9ecefc4f9971d36d166256a092663e60db711e7c887a2cac598d83aa31967ee"}
Sep 05 20:03:02.493666 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.493617    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" podStartSLOduration=1.49360277 podStartE2EDuration="1.49360277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:02.491867073 +0000 UTC m=+1191.908940501" watchObservedRunningTime="2026-09-05 20:03:02.49360277 +0000 UTC m=+1191.910676184"
Sep 05 20:03:02.583414 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.583373    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:02.588877 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.588852    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:02.769249 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.769148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:02.871900 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.871844    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:02.970496 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:02.970461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:03.472062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.472031    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:03.472917 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.472898    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:03:03.717672 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.717634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:03.803881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.803782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:03.870265 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.870232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:03.910178 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.910141    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"]
Sep 05 20:03:03.912208 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.912171    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-588081-pl7zq"]
Sep 05 20:03:03.933348 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:03.933318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:04.238780 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:04.238737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:05.044684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.044649    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a085b51-c3b3-47ca-b304-8c51a7ebc126" path="/var/lib/kubelet/pods/3a085b51-c3b3-47ca-b304-8c51a7ebc126/volumes"
Sep 05 20:03:05.799117 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.799092    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:03:05.907795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907705    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.907795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907743    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.907795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907768    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.907795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkpwm\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907825    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907849    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907869    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907917    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907990    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs\") pod \"ca950656-522c-4b8f-a2da-b45f21762f4a\" (UID: \"ca950656-522c-4b8f-a2da-b45f21762f4a\") "
Sep 05 20:03:05.908121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.907997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908239    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908363    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908410    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908444    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908374    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:05.908481 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.908463    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:05.910121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.910094    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm" (OuterVolumeSpecName: "kube-api-access-wkpwm") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "kube-api-access-wkpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:05.910262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.910150    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:05.910262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.910175    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:05.910262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:05.910214    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca950656-522c-4b8f-a2da-b45f21762f4a" (UID: "ca950656-522c-4b8f-a2da-b45f21762f4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:03:06.009107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009077    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009104    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009115    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009124    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkpwm\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-kube-api-access-wkpwm\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009134    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca950656-522c-4b8f-a2da-b45f21762f4a-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009144    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca950656-522c-4b8f-a2da-b45f21762f4a-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.009342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.009153    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca950656-522c-4b8f-a2da-b45f21762f4a-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:06.488134 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.488097    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerID="88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b" exitCode=0
Sep 05 20:03:06.488565 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.488144    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"
Sep 05 20:03:06.488565 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.488179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerDied","Data":"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"}
Sep 05 20:03:06.488565 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.488253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss" event={"ID":"ca950656-522c-4b8f-a2da-b45f21762f4a","Type":"ContainerDied","Data":"cd24eccbe479550ad17cd7e61ab7b057b14724780bdb80583686d3f6def5c11f"}
Sep 05 20:03:06.488565 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.488277    2575 scope.go:117] "RemoveContainer" containerID="88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"
Sep 05 20:03:06.497469 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.497453    2575 scope.go:117] "RemoveContainer" containerID="7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"
Sep 05 20:03:06.505264 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.505247    2575 scope.go:117] "RemoveContainer" containerID="88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"
Sep 05 20:03:06.505516 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:03:06.505494    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b\": container with ID starting with 88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b not found: ID does not exist" containerID="88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"
Sep 05 20:03:06.505593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.505527    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b"} err="failed to get container status \"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b\": rpc error: code = NotFound desc = could not find container \"88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b\": container with ID starting with 88815201df021a1aab6c65f07aeb82c054b2fb08d8d12c7a3bb5d0aa8a15e66b not found: ID does not exist"
Sep 05 20:03:06.505593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.505549    2575 scope.go:117] "RemoveContainer" containerID="7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"
Sep 05 20:03:06.505798 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:03:06.505780    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8\": container with ID starting with 7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8 not found: ID does not exist" containerID="7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"
Sep 05 20:03:06.505838 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.505805    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8"} err="failed to get container status \"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8\": rpc error: code = NotFound desc = could not find container \"7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8\": container with ID starting with 7a7b087196825de9de9fe30ff4eb7b7f612e47139c09f47918bd907620a227f8 not found: ID does not exist"
Sep 05 20:03:06.509501 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.509478    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:03:06.513343 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:06.513323    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-49af4f-openshift-default-78f666f4d8-snfss"]
Sep 05 20:03:07.049934 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:07.049894    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" path="/var/lib/kubelet/pods/ca950656-522c-4b8f-a2da-b45f21762f4a/volumes"
Sep 05 20:03:07.686292 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:07.686259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:07.786650 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:07.786610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:08.038246 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:08.038135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:03:08.341351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:08.341319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:10.158406 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.158377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:10.217870 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.217835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:10.396284 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.396253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:10.704716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.704680    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"]
Sep 05 20:03:10.705085 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705072    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.705144 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705089    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.705144 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705102    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.705144 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705110    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.705271 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705218    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.705271 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.705233    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca950656-522c-4b8f-a2da-b45f21762f4a" containerName="istio-proxy"
Sep 05 20:03:10.707157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.707142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:10.709837 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.709812    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:03:10.709955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.709856    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:03:10.715681 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.715656    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"]
Sep 05 20:03:10.754241 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.754180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc\" (UID: \"8be67237-74e9-42b7-b3f5-9adcc9c2211e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:10.855521 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.855488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc\" (UID: \"8be67237-74e9-42b7-b3f5-9adcc9c2211e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:10.856131 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.856110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc\" (UID: \"8be67237-74e9-42b7-b3f5-9adcc9c2211e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:10.897263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:10.897229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:11.020812 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.020730    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:03:11.022884 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.022862    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:03:11.025454 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.025437    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:03:11.027784 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.027766    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:11.306349 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.306318    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"]
Sep 05 20:03:11.308861 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:03:11.308836    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be67237_74e9_42b7_b3f5_9adcc9c2211e.slice/crio-7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0 WatchSource:0}: Error finding container 7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0: Status 404 returned error can't find the container with id 7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0
Sep 05 20:03:11.514517 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:11.514485    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc" event={"ID":"8be67237-74e9-42b7-b3f5-9adcc9c2211e","Type":"ContainerStarted","Data":"7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0"}
Sep 05 20:03:12.520584 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:12.520545    2575 generic.go:356] "Generic (PLEG): container finished" podID="8be67237-74e9-42b7-b3f5-9adcc9c2211e" containerID="e212b4caa61ba0e72697e9924f1e61bff3e6107a9e5a400cc4b7b21a15c9eb12" exitCode=0
Sep 05 20:03:12.520970 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:12.520629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc" event={"ID":"8be67237-74e9-42b7-b3f5-9adcc9c2211e","Type":"ContainerDied","Data":"e212b4caa61ba0e72697e9924f1e61bff3e6107a9e5a400cc4b7b21a15c9eb12"}
Sep 05 20:03:13.672354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:13.672327    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:13.784707 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:13.784601    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca\") pod \"8be67237-74e9-42b7-b3f5-9adcc9c2211e\" (UID: \"8be67237-74e9-42b7-b3f5-9adcc9c2211e\") "
Sep 05 20:03:13.784963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:13.784940    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8be67237-74e9-42b7-b3f5-9adcc9c2211e" (UID: "8be67237-74e9-42b7-b3f5-9adcc9c2211e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:13.885922 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:13.885883    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be67237-74e9-42b7-b3f5-9adcc9c2211e-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:14.376244 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.376201    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:14.432110 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.432073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:14.531849 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.531821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"
Sep 05 20:03:14.532010 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.531862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc" event={"ID":"8be67237-74e9-42b7-b3f5-9adcc9c2211e","Type":"ContainerDied","Data":"7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0"}
Sep 05 20:03:14.532010 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.531917    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b39aa976bb42016bc8df429eec300238eb81d144ed165b7c3d9a9d16e9d49d0"
Sep 05 20:03:14.601304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.601265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:14.875746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.875703    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:03:14.876226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.876212    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be67237-74e9-42b7-b3f5-9adcc9c2211e" containerName="revoke-keys"
Sep 05 20:03:14.876304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.876233    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be67237-74e9-42b7-b3f5-9adcc9c2211e" containerName="revoke-keys"
Sep 05 20:03:14.876394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.876365    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be67237-74e9-42b7-b3f5-9adcc9c2211e" containerName="revoke-keys"
Sep 05 20:03:14.878446 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.878423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.881041 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.881018    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fedc2772-openshift-default-dockercfg-rf8n8\""
Sep 05 20:03:14.892527 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.892500    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:03:14.910750 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.910727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:14.995485 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995676 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995676 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qpt\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995961 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995961 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:14.995961 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:14.995919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097002 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.096964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qpt\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097571 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097571 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097571 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097457    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097571 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.097894 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.097873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.099856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.099835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.100072 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.100050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.107852 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.107826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.108295 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.108276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qpt\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt\") pod \"e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") " pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.150411 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.150297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:15.189986 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.189955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:15.468499 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.468472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:03:15.470444 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:03:15.470416    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed16cd7_3f7d_43b1_9d1d_68184fae7609.slice/crio-563f3f25739a6a9d3719561adf0b04d090b7ec31c259c380e99acdc36860722d WatchSource:0}: Error finding container 563f3f25739a6a9d3719561adf0b04d090b7ec31c259c380e99acdc36860722d: Status 404 returned error can't find the container with id 563f3f25739a6a9d3719561adf0b04d090b7ec31c259c380e99acdc36860722d
Sep 05 20:03:15.473217 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.473167    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:15.473290 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.473250    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:15.473290 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.473281    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:15.539250 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.539215    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerStarted","Data":"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"}
Sep 05 20:03:15.539379 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.539308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerStarted","Data":"563f3f25739a6a9d3719561adf0b04d090b7ec31c259c380e99acdc36860722d"}
Sep 05 20:03:15.560030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:15.559959    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podStartSLOduration=1.5599395619999998 podStartE2EDuration="1.559939562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:15.5588017 +0000 UTC m=+1204.975875120" watchObservedRunningTime="2026-09-05 20:03:15.559939562 +0000 UTC m=+1204.977012986"
Sep 05 20:03:16.190266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:16.190230    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:16.194840 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:16.194815    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:16.544694 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:16.544609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:16.545745 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:16.545726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:17.505364 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:17.505325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:17.596815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:17.596775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:18.238968 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:18.238930    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:03:18.885990 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:18.885951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:18.977425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:18.977389    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:19.080741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:19.080705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:19.181208 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:19.181101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:19.393688 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:19.393651    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:19.499246 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:19.499139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:19.572794 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:19.572755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:20.869617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:20.869582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:21.413266 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.413232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:21.417702 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.417668    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"]
Sep 05 20:03:21.422239 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.422213    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1dc7b1-29810640-sx8pt"]
Sep 05 20:03:21.492355 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.492312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:21.562678 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.562643    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:21.676213 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.676119    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:21.704447 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.704412    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"]
Sep 05 20:03:21.714383 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.714344    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dc7b1-jxphc"]
Sep 05 20:03:21.748133 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:21.748103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:22.053574 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:22.053485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:23.043322 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:23.043292    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be67237-74e9-42b7-b3f5-9adcc9c2211e" path="/var/lib/kubelet/pods/8be67237-74e9-42b7-b3f5-9adcc9c2211e/volumes"
Sep 05 20:03:23.043605 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:23.043592    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffeb245-877f-49a3-8e3d-43642f220b83" path="/var/lib/kubelet/pods/fffeb245-877f-49a3-8e3d-43642f220b83/volumes"
Sep 05 20:03:28.030074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.030037    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:28.130539 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.130499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:28.278726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.277383    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 20:03:28.278726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.277656    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" containerID="cri-o://e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802" gracePeriod=30
Sep 05 20:03:28.304809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.303715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:28.731288 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.731249    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:03:28.807384 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.807343    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:28.886712 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.886662    2575 patch_prober.go:28] interesting pod/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 20:03:28.887030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:28.886987    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 20:03:29.111958 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:29.111911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:29.609432 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:29.609304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:31.889294 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:31.889258    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:31.973512 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:31.973479    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:32.586353 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:32.586314    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:03:32.586536 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:32.586386    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:03:33.147225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.147172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:33.247465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.247372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:33.313221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.313148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:33.531122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.531095    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 20:03:33.641868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.641768    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:33.647154 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.647124    2575 generic.go:356] "Generic (PLEG): container finished" podID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerID="3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5" exitCode=0
Sep 05 20:03:33.647154 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.647142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerDied","Data":"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"}
Sep 05 20:03:33.647671 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.647643    2575 scope.go:117] "RemoveContainer" containerID="3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"
Sep 05 20:03:33.648861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.648827    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:33.648952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.648910    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:33.649016 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.648952    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:33.649137 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.649120    2575 generic.go:356] "Generic (PLEG): container finished" podID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerID="e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802" exitCode=0
Sep 05 20:03:33.649207 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.649167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"
Sep 05 20:03:33.649255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.649209    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerDied","Data":"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"}
Sep 05 20:03:33.649255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.649247    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t" event={"ID":"d514b6d1-890c-4102-9e24-f792dfb73ae6","Type":"ContainerDied","Data":"4de59a4c449d41460a7d32759369dc2c490cac19a2732ec1349d8ba5db90566e"}
Sep 05 20:03:33.649320 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.649267    2575 scope.go:117] "RemoveContainer" containerID="e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"
Sep 05 20:03:33.660977 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.660955    2575 scope.go:117] "RemoveContainer" containerID="d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"
Sep 05 20:03:33.678851 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.678819    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679007 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.678884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66z4g\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679007 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.678926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679007 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.678950    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679007 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.678994    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679282 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679070    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679282 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679110    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:33.679282 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679263    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679452 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679452 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679375    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data\") pod \"d514b6d1-890c-4102-9e24-f792dfb73ae6\" (UID: \"d514b6d1-890c-4102-9e24-f792dfb73ae6\") "
Sep 05 20:03:33.679722 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.679701    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.680020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data" (OuterVolumeSpecName: "istio-data") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.680702    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.680800    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.680934    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.681084    2575 scope.go:117] "RemoveContainer" containerID="e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:03:33.681813    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802\": container with ID starting with e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802 not found: ID does not exist" containerID="e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.681846    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802"} err="failed to get container status \"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802\": rpc error: code = NotFound desc = could not find container \"e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802\": container with ID starting with e651cff9bc67d5db8ba657455d49903de841d1d38d669a0283cb8221714f9802 not found: ID does not exist"
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.681869    2575 scope.go:117] "RemoveContainer" containerID="d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:03:33.682288    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1\": container with ID starting with d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1 not found: ID does not exist" containerID="d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"
Sep 05 20:03:33.682354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.682320    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1"} err="failed to get container status \"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1\": rpc error: code = NotFound desc = could not find container \"d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1\": container with ID starting with d44a46f68ff5fdfa79a446592b90bd59153a12dffa9f6760541d0010d1bf4ce1 not found: ID does not exist"
Sep 05 20:03:33.683523 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.683496    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:33.683621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.683578    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:03:33.683621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.683607    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token" (OuterVolumeSpecName: "istio-token") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:33.683751 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.683687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g" (OuterVolumeSpecName: "kube-api-access-66z4g") pod "d514b6d1-890c-4102-9e24-f792dfb73ae6" (UID: "d514b6d1-890c-4102-9e24-f792dfb73ae6"). InnerVolumeSpecName "kube-api-access-66z4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:33.742730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.742686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:33.780774 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780734    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d514b6d1-890c-4102-9e24-f792dfb73ae6-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.780774 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780763    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.780774 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780776    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.781028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780789    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66z4g\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-kube-api-access-66z4g\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.781028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780804    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.781028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780816    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.781028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780826    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d514b6d1-890c-4102-9e24-f792dfb73ae6-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.781028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.780836    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d514b6d1-890c-4102-9e24-f792dfb73ae6-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:33.972851 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.972820    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 20:03:33.976335 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:33.976298    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1dc7b1-openshift-default-7d5dfcf8dd-6kq5t"]
Sep 05 20:03:34.655494 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:34.655457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerStarted","Data":"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"}
Sep 05 20:03:35.042579 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:35.042494    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" path="/var/lib/kubelet/pods/d514b6d1-890c-4102-9e24-f792dfb73ae6/volumes"
Sep 05 20:03:35.191226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:35.191165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:35.191474 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:35.191446    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 05 20:03:35.191557 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:35.191508    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 05 20:03:36.190659 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:36.190620    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 05 20:03:36.191295 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:36.191260    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 05 20:03:37.126778 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:37.126742    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:03:37.191519 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:37.191467    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 05 20:03:37.192018 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:37.191539    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 05 20:03:38.190573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:38.190526    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 05 20:03:38.190885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:38.190851    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 05 20:03:39.191726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:39.191254    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 05 20:03:39.191726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:39.191325    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 05 20:03:39.766519 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:39.766480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:40.193847 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:40.193817    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:40.194329 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:40.194097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:40.194755 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:40.194737    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:03:42.429997 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:42.429948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:43.065952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:43.065918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:43.163435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:43.163400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:43.487948 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:43.487857    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:46.247462 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:46.247420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:49.384101 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.384063    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:49.484613 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.484576    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:49.727917 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.727833    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"]
Sep 05 20:03:49.728248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.728234    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:49.728248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.728247    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:49.728335 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.728326    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:49.730418 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.730399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:49.733312 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.733290    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:03:49.733407 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.733293    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:03:49.743618 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.743594    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"]
Sep 05 20:03:49.827078 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.827052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj\" (UID: \"a341473b-b84c-4783-90e7-518aa27ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:49.928400 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.928365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj\" (UID: \"a341473b-b84c-4783-90e7-518aa27ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:49.928985 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:49.928965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj\" (UID: \"a341473b-b84c-4783-90e7-518aa27ba1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:50.041013 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:50.040925    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:50.310223 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:50.310177    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"]
Sep 05 20:03:50.312551 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:03:50.312523    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda341473b_b84c_4783_90e7_518aa27ba1fb.slice/crio-7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77 WatchSource:0}: Error finding container 7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77: Status 404 returned error can't find the container with id 7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77
Sep 05 20:03:50.744160 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:50.744121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj" event={"ID":"a341473b-b84c-4783-90e7-518aa27ba1fb","Type":"ContainerStarted","Data":"79f8d841ed172bbcf822743566ee6ec89ea2228231c2b87a2d4f3120f247a692"}
Sep 05 20:03:50.744585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:50.744232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj" event={"ID":"a341473b-b84c-4783-90e7-518aa27ba1fb","Type":"ContainerStarted","Data":"7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77"}
Sep 05 20:03:50.761849 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:50.761791    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj" podStartSLOduration=1.7617704349999999 podStartE2EDuration="1.761770435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:50.761391142 +0000 UTC m=+1240.178464565" watchObservedRunningTime="2026-09-05 20:03:50.761770435 +0000 UTC m=+1240.178843858"
Sep 05 20:03:51.750227 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:51.750178    2575 generic.go:356] "Generic (PLEG): container finished" podID="a341473b-b84c-4783-90e7-518aa27ba1fb" containerID="79f8d841ed172bbcf822743566ee6ec89ea2228231c2b87a2d4f3120f247a692" exitCode=0
Sep 05 20:03:51.750628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:51.750286    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj" event={"ID":"a341473b-b84c-4783-90e7-518aa27ba1fb","Type":"ContainerDied","Data":"79f8d841ed172bbcf822743566ee6ec89ea2228231c2b87a2d4f3120f247a692"}
Sep 05 20:03:52.876905 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:52.876880    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:52.963034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:52.963002    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca\") pod \"a341473b-b84c-4783-90e7-518aa27ba1fb\" (UID: \"a341473b-b84c-4783-90e7-518aa27ba1fb\") "
Sep 05 20:03:52.963368 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:52.963344    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a341473b-b84c-4783-90e7-518aa27ba1fb" (UID: "a341473b-b84c-4783-90e7-518aa27ba1fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:53.063984 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:53.063955    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a341473b-b84c-4783-90e7-518aa27ba1fb-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:03:53.761767 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:53.761734    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"
Sep 05 20:03:53.761947 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:53.761777    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj" event={"ID":"a341473b-b84c-4783-90e7-518aa27ba1fb","Type":"ContainerDied","Data":"7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77"}
Sep 05 20:03:53.761947 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:53.761819    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b709dc7de035a27e478422d903d10fa49affd38ea954ec550c11f56da513a77"
Sep 05 20:03:54.991539 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:54.991493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:56.840302 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:56.840264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:57.150516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.150437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:57.411844 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.411757    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:03:57.412159 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412146    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:57.412221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412161    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:57.412221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412169    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a341473b-b84c-4783-90e7-518aa27ba1fb" containerName="revoke-keys"
Sep 05 20:03:57.412221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412175    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a341473b-b84c-4783-90e7-518aa27ba1fb" containerName="revoke-keys"
Sep 05 20:03:57.412332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412270    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d514b6d1-890c-4102-9e24-f792dfb73ae6" containerName="istio-proxy"
Sep 05 20:03:57.412332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.412282    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a341473b-b84c-4783-90e7-518aa27ba1fb" containerName="revoke-keys"
Sep 05 20:03:57.414833 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.414808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.421896 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.421871    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0cc31a-openshift-default-dockercfg-jxckl\""
Sep 05 20:03:57.429585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.429561    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:03:57.452094 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.452055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:57.502104 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rgh\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.502587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.502521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603288 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rgh\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.603815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.604012 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.604012 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.603988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.604233 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.604057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.604233 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.604079    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.604233 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.604126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.605988 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.605965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.606326 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.606308    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.611777 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.611749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.611885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.611817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rgh\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh\") pod \"e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") " pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:57.618632 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.618597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:03:57.726418 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:57.726320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:58.004885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.004856    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:03:58.007449 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:03:58.007422    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9febecf_a2dc_41cf_b397_8dadc7519b55.slice/crio-5c414fc10880d0d3982cc743cac128317a243718e8b6c74b8a7cf1c441516e7c WatchSource:0}: Error finding container 5c414fc10880d0d3982cc743cac128317a243718e8b6c74b8a7cf1c441516e7c: Status 404 returned error can't find the container with id 5c414fc10880d0d3982cc743cac128317a243718e8b6c74b8a7cf1c441516e7c
Sep 05 20:03:58.010089 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.010056    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:58.010174 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.010128    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:58.010174 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.010159    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:03:58.792035 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.792003    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerStarted","Data":"5c8acf6b3ae4776197c1aa38c6434775f0ec28ef390485bf2e4a8b7a2a898328"}
Sep 05 20:03:58.792180 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.792048    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerStarted","Data":"5c414fc10880d0d3982cc743cac128317a243718e8b6c74b8a7cf1c441516e7c"}
Sep 05 20:03:58.816729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:58.816671    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podStartSLOduration=1.816655558 podStartE2EDuration="1.816655558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:58.814972437 +0000 UTC m=+1248.232045861" watchObservedRunningTime="2026-09-05 20:03:58.816655558 +0000 UTC m=+1248.233728982"
Sep 05 20:03:59.726993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:59.726949    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:59.731909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:59.731883    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:59.797673 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:59.797643    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:59.798732 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:59.798708    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:03:59.892283 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:03:59.892248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.440679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.440648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.511620 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.511589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.573969 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.573934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.611802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.611756    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"]
Sep 05 20:04:00.616912 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.616882    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fedc2772-ft2qj"]
Sep 05 20:04:00.675696 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.675663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.756642 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.756551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:00.956459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:00.956398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:01.047125 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:01.047039    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a341473b-b84c-4783-90e7-518aa27ba1fb" path="/var/lib/kubelet/pods/a341473b-b84c-4783-90e7-518aa27ba1fb/volumes"
Sep 05 20:04:01.727956 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:01.727925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:04:01.833321 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:01.833287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:01.941729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:01.941692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:02.161466 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:02.161431    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:02.333675 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:02.333642    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:02.434547 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:02.434444    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:06.900301 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:06.900180    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:06.958388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:06.958328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:07.121526 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.121493    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:04:07.121736 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.121713    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" containerID="cri-o://8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155" gracePeriod=30
Sep 05 20:04:07.142913 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.142875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:07.483361 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.483324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:07.584138 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.584104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:07.683040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:07.683008    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:10.197555 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:10.197515    2575 patch_prober.go:28] interesting pod/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 20:04:10.197959 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:10.197601    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 20:04:11.930111 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:11.930063    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:12.030929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.030885    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:04:12.031225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.031178    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" podUID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" containerName="istio-proxy" containerID="cri-o://e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa" gracePeriod=30
Sep 05 20:04:12.036225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.036199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:12.257466 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.257435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:12.362432 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.362383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:12.405746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.405715    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:04:12.548962 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.548875    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.548962 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.548916    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.548962 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.548958    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.548985    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549013    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99qpt\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549074    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549108    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549151    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549253    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo\") pod \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\" (UID: \"2ed16cd7-3f7d-43b1-9d1d-68184fae7609\") "
Sep 05 20:04:12.549578 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549281    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:12.549578 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:12.549578 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549473    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:12.549578 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549521    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:12.549795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549647    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.549795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549661    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.549795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549669    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.549795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549666    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:04:12.549795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.549685    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.551275 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.551243    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:12.551396 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.551301    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:12.551455 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.551391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt" (OuterVolumeSpecName: "kube-api-access-99qpt") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "kube-api-access-99qpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:12.551522 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.551451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ed16cd7-3f7d-43b1-9d1d-68184fae7609" (UID: "2ed16cd7-3f7d-43b1-9d1d-68184fae7609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:04:12.586388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.586350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:12.650856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.650823    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.650856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.650851    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qpt\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-kube-api-access-99qpt\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.650856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.650862    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.651090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.650870    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.651090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.650878    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed16cd7-3f7d-43b1-9d1d-68184fae7609-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:12.867866 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.867829    2575 generic.go:356] "Generic (PLEG): container finished" podID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerID="8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155" exitCode=0
Sep 05 20:04:12.868040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.867884    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"
Sep 05 20:04:12.868040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.867919    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerDied","Data":"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"}
Sep 05 20:04:12.868040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.867983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477" event={"ID":"2ed16cd7-3f7d-43b1-9d1d-68184fae7609","Type":"ContainerDied","Data":"563f3f25739a6a9d3719561adf0b04d090b7ec31c259c380e99acdc36860722d"}
Sep 05 20:04:12.868040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.868015    2575 scope.go:117] "RemoveContainer" containerID="8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"
Sep 05 20:04:12.888613 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.888588    2575 scope.go:117] "RemoveContainer" containerID="3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"
Sep 05 20:04:12.902657 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.902622    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:04:12.904856 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.904832    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fedc2772-openshift-default-6478bbcc56-6w477"]
Sep 05 20:04:12.905868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.905848    2575 scope.go:117] "RemoveContainer" containerID="8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"
Sep 05 20:04:12.906174 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:12.906154    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155\": container with ID starting with 8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155 not found: ID does not exist" containerID="8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"
Sep 05 20:04:12.906240 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.906196    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155"} err="failed to get container status \"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155\": rpc error: code = NotFound desc = could not find container \"8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155\": container with ID starting with 8b40369b0a45941fd2a2ce324aeab0d2ca1370947229d0d8a61cebb6da05d155 not found: ID does not exist"
Sep 05 20:04:12.906240 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.906223    2575 scope.go:117] "RemoveContainer" containerID="3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"
Sep 05 20:04:12.906538 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:12.906512    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5\": container with ID starting with 3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5 not found: ID does not exist" containerID="3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"
Sep 05 20:04:12.906593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:12.906544    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5"} err="failed to get container status \"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5\": rpc error: code = NotFound desc = could not find container \"3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5\": container with ID starting with 3e20179f8c6819b5db1437a487f5e433e74f771d754cec46addf047c65743bf5 not found: ID does not exist"
Sep 05 20:04:13.042795 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.042756    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" path="/var/lib/kubelet/pods/2ed16cd7-3f7d-43b1-9d1d-68184fae7609/volumes"
Sep 05 20:04:13.598752 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.598722    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:13.661889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.661854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:13.723921 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.723882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:13.827729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.827687    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:04:13.914721 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:13.914634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:17.279752 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.279724    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:04:17.384534 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.384507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:17.402606 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402577    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402749 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402640    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402749 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402670    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402749 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402701    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402749 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402733    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402964 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402805    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.402964 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402938    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:17.403068 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.402977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:17.403116 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403066    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.403116 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403087    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:17.403229 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403115    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.403229 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94xvl\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl\") pod \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\" (UID: \"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55\") "
Sep 05 20:04:17.403428 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403359    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:17.403428 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403409    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:04:17.403628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403555    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.403628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403570    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.403628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403579    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.403628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403588    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.403628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.403599    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.405399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.405374    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:17.405399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.405383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:04:17.405615 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.405596    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:17.405685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.405597    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl" (OuterVolumeSpecName: "kube-api-access-94xvl") pod "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" (UID: "4d95f0c7-3f34-4d53-9449-36cfc6ee2f55"). InnerVolumeSpecName "kube-api-access-94xvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:17.504939 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.504893    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.504939 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.504925    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94xvl\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-kube-api-access-94xvl\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.504939 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.504936    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.504939 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.504944    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:04:17.906741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.906700    2575 generic.go:356] "Generic (PLEG): container finished" podID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" containerID="e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa" exitCode=0
Sep 05 20:04:17.906946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.906744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" event={"ID":"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55","Type":"ContainerDied","Data":"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"}
Sep 05 20:04:17.906946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.906801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh" event={"ID":"4d95f0c7-3f34-4d53-9449-36cfc6ee2f55","Type":"ContainerDied","Data":"b9ecefc4f9971d36d166256a092663e60db711e7c887a2cac598d83aa31967ee"}
Sep 05 20:04:17.906946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.906824    2575 scope.go:117] "RemoveContainer" containerID="e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"
Sep 05 20:04:17.906946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.906752    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"
Sep 05 20:04:17.916541 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.916523    2575 scope.go:117] "RemoveContainer" containerID="e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"
Sep 05 20:04:17.916815 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:17.916798    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa\": container with ID starting with e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa not found: ID does not exist" containerID="e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"
Sep 05 20:04:17.916876 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.916821    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa"} err="failed to get container status \"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa\": rpc error: code = NotFound desc = could not find container \"e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa\": container with ID starting with e8f43a178dd6652bdd16b8f14527154b3b9a33382b276cd1e2b4c6ca3333a8fa not found: ID does not exist"
Sep 05 20:04:17.930155 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.930131    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:04:17.934323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:17.934300    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-0741a0-openshift-default-6954f9fb58-ljgjh"]
Sep 05 20:04:18.915197 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.915161    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerID="5c8acf6b3ae4776197c1aa38c6434775f0ec28ef390485bf2e4a8b7a2a898328" exitCode=0
Sep 05 20:04:18.915690 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.915239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerDied","Data":"5c8acf6b3ae4776197c1aa38c6434775f0ec28ef390485bf2e4a8b7a2a898328"}
Sep 05 20:04:18.915690 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.915625    2575 scope.go:117] "RemoveContainer" containerID="5c8acf6b3ae4776197c1aa38c6434775f0ec28ef390485bf2e4a8b7a2a898328"
Sep 05 20:04:18.916588 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.916561    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:18.916645 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.916631    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:18.916685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:18.916665    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:19.043840 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:19.043801    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" path="/var/lib/kubelet/pods/4d95f0c7-3f34-4d53-9449-36cfc6ee2f55/volumes"
Sep 05 20:04:19.830227 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:19.830171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:19.923563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:19.923530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerStarted","Data":"2b5e19c05528619b71a2dde3fc44da9cf5ab72b3f0499219dd79aca865cd873a"}
Sep 05 20:04:19.927712 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:19.927686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:20.682911 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.682863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:20.727346 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.727304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:04:20.727543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.727521    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 05 20:04:20.727595 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.727568    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 05 20:04:20.962364 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962288    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:04:20.962830 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962816    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.962877 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962833    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.962877 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962853    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.962877 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962864    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.962969 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962900    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" containerName="istio-proxy"
Sep 05 20:04:20.962969 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962909    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" containerName="istio-proxy"
Sep 05 20:04:20.963038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.962988    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.963038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.963003    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d95f0c7-3f34-4d53-9449-36cfc6ee2f55" containerName="istio-proxy"
Sep 05 20:04:20.963038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.963015    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed16cd7-3f7d-43b1-9d1d-68184fae7609" containerName="istio-proxy"
Sep 05 20:04:20.965629 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.965610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:20.968306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.968284    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1436b9-openshift-default-dockercfg-9bq46\""
Sep 05 20:04:20.995208 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.987024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:04:20.999244 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:20.998042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:21.139741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.139909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.139909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.139909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.139909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.139909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mskxs\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.140103 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.140103 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.140103 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.139985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mskxs\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.240807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241026 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241026 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241026 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.240989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241221 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.241177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241296 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.241265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241402 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.241382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.241516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.241495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.242907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.242884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.243355 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.243332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.249843 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.249821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.250043 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.250021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mskxs\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs\") pod \"e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") " pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.287441 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.287414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:21.575532 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.575497    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:04:21.587138 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:04:21.587097    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5083df0_988b_497c_8866_c4bf1ce77d94.slice/crio-b3d9a3c6c61746a02b5c0e0c183f975c3f05eb88707341882fcbc556ed6c9f7d WatchSource:0}: Error finding container b3d9a3c6c61746a02b5c0e0c183f975c3f05eb88707341882fcbc556ed6c9f7d: Status 404 returned error can't find the container with id b3d9a3c6c61746a02b5c0e0c183f975c3f05eb88707341882fcbc556ed6c9f7d
Sep 05 20:04:21.590246 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.590200    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:21.590354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.590295    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:21.590354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.590339    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:21.726957 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.726921    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 05 20:04:21.727124 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.726990    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 05 20:04:21.834990 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.834901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:21.935318 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.935285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerStarted","Data":"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"}
Sep 05 20:04:21.935455 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:21.935332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerStarted","Data":"b3d9a3c6c61746a02b5c0e0c183f975c3f05eb88707341882fcbc556ed6c9f7d"}
Sep 05 20:04:22.288625 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.288479    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:22.299717 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.299686    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:22.321526 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.321469    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podStartSLOduration=2.321452341 podStartE2EDuration="2.321452341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:21.98311309 +0000 UTC m=+1271.400186514" watchObservedRunningTime="2026-09-05 20:04:22.321452341 +0000 UTC m=+1271.738525764"
Sep 05 20:04:22.726952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.726911    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 05 20:04:22.727167 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.726974    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 05 20:04:22.940715 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.940688    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:22.941731 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:22.941713    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:23.726941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:23.726900    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 05 20:04:23.727344 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:23.726972    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 05 20:04:23.912809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:23.912775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:24.013028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:24.012944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:24.727732 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:24.727695    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 05 20:04:24.728127 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:24.727754    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 05 20:04:25.178096 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.178061    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:04:25.181048 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.181024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.183999 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.183975    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 05 20:04:25.185225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.185202    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:04:25.185429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.185409    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-1436b9-serving-cert\""
Sep 05 20:04:25.185641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.185624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 20:04:25.185925 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.185905    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 05 20:04:25.194247 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.194218    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:04:25.207653 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.207622    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:25.282345 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.282306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.282546 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.282366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rrz\" (UniqueName: \"kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.282546 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.282460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.282663 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.282564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.305591 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.305550    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:25.383857 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.383817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rrz\" (UniqueName: \"kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.384057 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.383866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.384057 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.383934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.384057 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.383988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.384244 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:25.384105    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-serving-cert: secret "maas-api-e2e-shared-a-w2-1436b9-serving-cert" not found
Sep 05 20:04:25.384244 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:25.384204    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls podName:ec84bc48-c167-4002-b8c0-49da6533818f nodeName:}" failed. No retries permitted until 2026-09-05 20:04:25.884166932 +0000 UTC m=+1275.301240351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" (UID: "ec84bc48-c167-4002-b8c0-49da6533818f") : secret "maas-api-e2e-shared-a-w2-1436b9-serving-cert" not found
Sep 05 20:04:25.384702 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.384682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.386413 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.386390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.392787 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.392765    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rrz\" (UniqueName: \"kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.618851 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.618819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:25.731088 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.731048    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:04:25.731587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.731476    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:04:25.732042 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.732023    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:04:25.787478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.787440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:25.884715 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.884626    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:25.896917 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.896882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.899654 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.899626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:25.958728 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:25.958696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:26.095781 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.095749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:26.367474 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.367449    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:04:26.369585 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:04:26.369560    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec84bc48_c167_4002_b8c0_49da6533818f.slice/crio-1ac0889f746499d9d95e642a83b8629dcdb22a1934d54e50a7e625989d4882f1 WatchSource:0}: Error finding container 1ac0889f746499d9d95e642a83b8629dcdb22a1934d54e50a7e625989d4882f1: Status 404 returned error can't find the container with id 1ac0889f746499d9d95e642a83b8629dcdb22a1934d54e50a7e625989d4882f1
Sep 05 20:04:26.963152 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.963055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" event={"ID":"ec84bc48-c167-4002-b8c0-49da6533818f","Type":"ContainerStarted","Data":"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"}
Sep 05 20:04:26.963152 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.963119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" event={"ID":"ec84bc48-c167-4002-b8c0-49da6533818f","Type":"ContainerStarted","Data":"1ac0889f746499d9d95e642a83b8629dcdb22a1934d54e50a7e625989d4882f1"}
Sep 05 20:04:26.963585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.963292    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:26.980429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:26.980379    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" podStartSLOduration=1.6531337640000001 podStartE2EDuration="1.980362692s" totalImagesPullingTime="327.228928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:26.979637306 +0000 UTC m=+1276.396710728" watchObservedRunningTime="2026-09-05 20:04:26.980362692 +0000 UTC m=+1276.397436115"
Sep 05 20:04:30.531553 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:30.531512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:04:32.978474 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:32.978438    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:04:38.983334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:38.983293    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:04:38.983810 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:38.983357    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:04:41.044736 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.044702    2575 generic.go:356] "Generic (PLEG): container finished" podID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerID="6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d" exitCode=0
Sep 05 20:04:41.045156 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.044774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerDied","Data":"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"}
Sep 05 20:04:41.045222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.045157    2575 scope.go:117] "RemoveContainer" containerID="6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"
Sep 05 20:04:41.046179 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.046148    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:41.046317 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.046245    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:41.046317 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:41.046293    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:42.051826 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:42.051794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerStarted","Data":"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"}
Sep 05 20:04:42.288726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:42.288688    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:42.289068 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:42.289043    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 05 20:04:42.289526 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:42.289235    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 05 20:04:43.288177 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:43.288130    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 05 20:04:43.288721 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:43.288224    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 05 20:04:43.447725 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:43.447683    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:44.288065 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:44.288022    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 05 20:04:44.288428 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:44.288392    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 05 20:04:45.288699 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:45.288655    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 05 20:04:45.289331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:45.289294    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 05 20:04:46.288020 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:46.287972    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 05 20:04:46.288405 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:46.288371    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 05 20:04:47.291892 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:47.291861    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:47.292287 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:47.292158    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:47.292819 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:47.292796    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:04:47.533305 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:47.533262    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:04:48.807765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:48.807734    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:49.118991 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.118893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:49.383869 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.383779    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:04:49.386487 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.386469    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.389201 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.389162    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ca4fba-openshift-default-dockercfg-6ssmm\""
Sep 05 20:04:49.398475 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.398440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:04:49.404417 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.404386    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:49.474406 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.474371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:49.546683 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.546883 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.546883 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.546883 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.547091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2kv\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.547091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.546956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.547091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.547009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.547091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.547078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.547351 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.547134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648672 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2kv\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648672 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648672 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.648907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.648813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.649147 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.649019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.649147 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.649101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.649323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.649305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.650787 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.650761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.651104 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.651087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.657623 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.657592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.657864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.657848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2kv\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv\") pod \"e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") " pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.697798 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.697752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:49.973860 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.973832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:04:49.975808 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:04:49.975779    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c18c5bf_2369_4cb2_9dae_956abc0a82b0.slice/crio-4ab3c5cf4e113d5d461bdae88e354f9c6d549fb5dbfbaa8fc3efe6f2187dee8a WatchSource:0}: Error finding container 4ab3c5cf4e113d5d461bdae88e354f9c6d549fb5dbfbaa8fc3efe6f2187dee8a: Status 404 returned error can't find the container with id 4ab3c5cf4e113d5d461bdae88e354f9c6d549fb5dbfbaa8fc3efe6f2187dee8a
Sep 05 20:04:49.978219 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.978161    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:49.978280 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.978265    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:49.978313 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:49.978298    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:04:50.102998 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:50.102957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerStarted","Data":"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"}
Sep 05 20:04:50.103171 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:50.103028    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerStarted","Data":"4ab3c5cf4e113d5d461bdae88e354f9c6d549fb5dbfbaa8fc3efe6f2187dee8a"}
Sep 05 20:04:50.123967 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:50.123913    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podStartSLOduration=1.123892552 podStartE2EDuration="1.123892552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:50.122718464 +0000 UTC m=+1299.539791891" watchObservedRunningTime="2026-09-05 20:04:50.123892552 +0000 UTC m=+1299.540965975"
Sep 05 20:04:50.697854 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:50.697803    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:51.703207 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:51.703165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:51.995166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:51.995071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:52.097504 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:52.097472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:52.112955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:52.112928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:52.113759 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:52.113739    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:04:53.109950 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.109912    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:04:53.112489 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.112467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.115817 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.115612    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ca4fba-serving-cert\""
Sep 05 20:04:53.126332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.126302    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:04:53.149046 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.149009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:53.178254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.178216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.178429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.178288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.178429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.178314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbt6l\" (UniqueName: \"kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.178429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.178330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.246627 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.246591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:53.279725 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.279692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.279866 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.279755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbt6l\" (UniqueName: \"kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.279866 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.279787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.280001 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.279936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.280106 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:53.280090    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-serving-cert: secret "maas-api-e2e-shared-b-w2-ca4fba-serving-cert" not found
Sep 05 20:04:53.280173 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:04:53.280149    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls podName:33ec1803-6b7f-472a-b027-a5b132efdf61 nodeName:}" failed. No retries permitted until 2026-09-05 20:04:53.780131654 +0000 UTC m=+1303.197205061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" (UID: "33ec1803-6b7f-472a-b027-a5b132efdf61") : secret "maas-api-e2e-shared-b-w2-ca4fba-serving-cert" not found
Sep 05 20:04:53.280477 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.280454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.282227 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.282201    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.287988 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.287964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbt6l\" (UniqueName: \"kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.614622 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.614585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:53.708303 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.708272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:53.784745 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.784711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:53.787065 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:53.787034    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:54.026733 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:54.026643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:54.300248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:54.300213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:04:54.302692 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:04:54.302663    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ec1803_6b7f_472a_b027_a5b132efdf61.slice/crio-946ce8ba8bc78029afca9475d4ce505a34229202abd64fcb0db34d3921bb3dae WatchSource:0}: Error finding container 946ce8ba8bc78029afca9475d4ce505a34229202abd64fcb0db34d3921bb3dae: Status 404 returned error can't find the container with id 946ce8ba8bc78029afca9475d4ce505a34229202abd64fcb0db34d3921bb3dae
Sep 05 20:04:55.050748 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:55.050665    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:04:55.131701 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:55.131668    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" event={"ID":"33ec1803-6b7f-472a-b027-a5b132efdf61","Type":"ContainerStarted","Data":"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"}
Sep 05 20:04:55.131867 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:55.131719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" event={"ID":"33ec1803-6b7f-472a-b027-a5b132efdf61","Type":"ContainerStarted","Data":"946ce8ba8bc78029afca9475d4ce505a34229202abd64fcb0db34d3921bb3dae"}
Sep 05 20:04:55.131867 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:55.131805    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:04:55.151894 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:55.149207    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" podStartSLOduration=1.8376426750000001 podStartE2EDuration="2.149166892s" totalImagesPullingTime="311.524217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:55.148023984 +0000 UTC m=+1304.565097408" watchObservedRunningTime="2026-09-05 20:04:55.149166892 +0000 UTC m=+1304.566240316"
Sep 05 20:04:58.734028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:04:58.733985    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:05:01.143159 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:01.143133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:05:08.154362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:08.154318    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:05:08.154757 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:08.154403    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:05:08.717661 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:08.717628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:09.212465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.212429    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerID="12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5" exitCode=0
Sep 05 20:05:09.212943 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.212505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerDied","Data":"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"}
Sep 05 20:05:09.213068 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.213051    2575 scope.go:117] "RemoveContainer" containerID="12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"
Sep 05 20:05:09.214157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.214130    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:09.214257 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.214217    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:09.214257 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:09.214251    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:10.219626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:10.219589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerStarted","Data":"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"}
Sep 05 20:05:10.698308 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:10.698270    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:05:10.698505 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:10.698434    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 05 20:05:10.698557 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:10.698510    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 05 20:05:11.698971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:11.698932    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 05 20:05:11.699450 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:11.699025    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 05 20:05:12.698844 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:12.698804    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 05 20:05:12.699022 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:12.698864    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 05 20:05:13.698647 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:13.698610    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 05 20:05:13.698865 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:13.698674    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 05 20:05:14.698792 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:14.698748    2575 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 05 20:05:14.699289 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:14.698818    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 05 20:05:15.549974 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:15.549934    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:05:15.702077 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:15.702046    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:05:15.702487 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:15.702399    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:05:15.703009 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:15.702992    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:05:16.928020 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:16.927975    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:27.828394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:27.828359    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:05:29.986391 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:29.986362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:30.259685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.259600    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:05:30.265404 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.265381    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.268070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.268050    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ada83762-openshift-default-dockercfg-tvjx8\""
Sep 05 20:05:30.277104 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.277078    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:05:30.362253 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362446 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362446 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362446 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcfd\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.362823 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.362658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.463621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.463621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.463832 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.463890 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.463959 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcfd\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464037 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.463959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464037 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464335 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464300    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464421 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464368    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464536 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464663 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464587    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.464920 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.464897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.466084 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.466064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.466549 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.466528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.472065 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.472046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcfd\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.472222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.472180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token\") pod \"e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") " pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.576605 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.576567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:30.854945 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.854918    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:05:30.857220 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:05:30.857158    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7e80d9_c9e0_4ec0_ac85_4fba08533d66.slice/crio-4e499d2ae66771a7447abde875715865edf2f518c8e8cc70244f5a34c0f5f9ad WatchSource:0}: Error finding container 4e499d2ae66771a7447abde875715865edf2f518c8e8cc70244f5a34c0f5f9ad: Status 404 returned error can't find the container with id 4e499d2ae66771a7447abde875715865edf2f518c8e8cc70244f5a34c0f5f9ad
Sep 05 20:05:30.859437 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.859401    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:30.859535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.859466    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:30.859535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:30.859496    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:31.101946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.101905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:31.333630 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.333588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerStarted","Data":"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"}
Sep 05 20:05:31.333839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.333679    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerStarted","Data":"4e499d2ae66771a7447abde875715865edf2f518c8e8cc70244f5a34c0f5f9ad"}
Sep 05 20:05:31.357628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.357572    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podStartSLOduration=1.357554591 podStartE2EDuration="1.357554591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:05:31.356344238 +0000 UTC m=+1340.773417660" watchObservedRunningTime="2026-09-05 20:05:31.357554591 +0000 UTC m=+1340.774628014"
Sep 05 20:05:31.577645 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.577602    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:31.582862 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:31.582834    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:32.339594 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:32.339566    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:32.340648 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:32.340624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:33.245666 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:33.245630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:33.329475 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:33.329396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:34.386868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:34.386836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:34.484920 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:34.484880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:34.774088 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:34.773989    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:34.880382 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:34.880351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:34.974533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:34.974497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:48.380377 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:48.380341    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:05:48.380745 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:48.380412    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:05:48.728628 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:48.728546    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:05:49.443988 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.443952    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerID="4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162" exitCode=0
Sep 05 20:05:49.444542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.444027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerDied","Data":"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"}
Sep 05 20:05:49.444542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.444498    2575 scope.go:117] "RemoveContainer" containerID="4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"
Sep 05 20:05:49.445782 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.445755    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:49.445861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.445822    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:49.445861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:49.445854    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:05:50.451679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:50.451637    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerStarted","Data":"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"}
Sep 05 20:05:50.577081 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:50.577026    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:50.577420 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:50.577376    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:50.577498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:50.577463    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:51.577820 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:51.577778    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:51.578459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:51.578426    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:52.098054 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:52.098013    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:52.577498 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:52.577444    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:52.577716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:52.577531    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:53.145398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:53.145358    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:53.577484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:53.577445    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:53.577794 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:53.577764    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:54.577390 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:54.577353    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:54.578007 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:54.577972    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:55.251459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:55.251424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:05:55.577442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:55.577406    2575 patch_prober.go:28] interesting pod/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 05 20:05:55.577850 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:55.577460    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 05 20:05:56.580720 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:56.580687    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:56.581119 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:56.581077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:56.581818 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:56.581800    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:05:58.137714 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:05:58.137679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:05.951591 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:05.951558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:06:10.550435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:10.550397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:11.155699 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:11.155663    2575 scope.go:117] "RemoveContainer" containerID="a4e76831c23dcc8c0835aa6f9b15820070b69658d13f3d5876d936ca6f2881d5"
Sep 05 20:06:25.123572 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:25.123538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:06:42.064365 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:42.064319    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:06:57.933403 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:57.933361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:58.001363 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:58.001328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:58.098352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:58.098321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:58.158968 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:58.158937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:06:59.433507 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:06:59.433474    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:07:01.110123 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:01.110084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:06.140333 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:06.140239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:11.175251 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:11.175217    2575 scope.go:117] "RemoveContainer" containerID="323bd216a32983b3259c2363c0500c23156d092eea743f2fa0ef27fcb558c56e"
Sep 05 20:07:15.322668 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:15.322630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:15.923984 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:15.923954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:07:16.521553 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:16.521519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:25.086535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:25.086480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:25.792935 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:25.792894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:28.232581 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:28.232545    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:07:29.388241 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:29.388209    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:40.037154 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:40.037120    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:07:42.627886 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:42.627853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:43.668194 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:43.668139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:43.889088 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:43.889056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:43.987670 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:43.987566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:50.191492 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:50.191405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:50.292376 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:50.292347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:50.463047 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:50.462962    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:07:50.463252 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:50.463220    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy" containerID="cri-o://60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72" gracePeriod=30
Sep 05 20:07:50.483562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:50.483525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:51.829575 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:51.829536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:51.927435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:51.927403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:52.027436 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:52.027395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:52.700448 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:52.700413    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:52.799383 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:52.799344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:52.899360 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:52.899331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:54.226625 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:54.226593    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:07:55.686592 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.686555    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:07:55.709537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.709516    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:07:55.840965 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.840937    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.840973    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.840995    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841026    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpcfd\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841045    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841112    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841140    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841210    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841236    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs\") pod \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\" (UID: \"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66\") "
Sep 05 20:07:55.841516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:55.841516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841319    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:55.841716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841542    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:55.841716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841562    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:55.841716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841599    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.841716 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841639    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.841926 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.841729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:07:55.843409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.843377    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:07:55.843409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.843374    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:55.843552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.843419    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd" (OuterVolumeSpecName: "kube-api-access-fpcfd") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "kube-api-access-fpcfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:07:55.843552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.843471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" (UID: "3a7e80d9-c9e0-4ec0-ac85-4fba08533d66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:07:55.942641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942613    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942638    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942646    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942654    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942661    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942671    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpcfd\" (UniqueName: \"kubernetes.io/projected/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-kube-api-access-fpcfd\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:55.942933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:55.942680    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:07:56.112196 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.112085    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerID="60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72" exitCode=0
Sep 05 20:07:56.112196 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.112139    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"
Sep 05 20:07:56.112196 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.112161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerDied","Data":"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"}
Sep 05 20:07:56.112488 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.112238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6" event={"ID":"3a7e80d9-c9e0-4ec0-ac85-4fba08533d66","Type":"ContainerDied","Data":"4e499d2ae66771a7447abde875715865edf2f518c8e8cc70244f5a34c0f5f9ad"}
Sep 05 20:07:56.112488 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.112262    2575 scope.go:117] "RemoveContainer" containerID="60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"
Sep 05 20:07:56.121546 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.121528    2575 scope.go:117] "RemoveContainer" containerID="4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"
Sep 05 20:07:56.129757 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.129741    2575 scope.go:117] "RemoveContainer" containerID="60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"
Sep 05 20:07:56.130013 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:07:56.129997    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72\": container with ID starting with 60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72 not found: ID does not exist" containerID="60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"
Sep 05 20:07:56.130051 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.130022    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72"} err="failed to get container status \"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72\": rpc error: code = NotFound desc = could not find container \"60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72\": container with ID starting with 60eb086c02255986247a2e0f55ce51cd41e4aa4936e3b0a38745be16bbe6dc72 not found: ID does not exist"
Sep 05 20:07:56.130051 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.130037    2575 scope.go:117] "RemoveContainer" containerID="4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"
Sep 05 20:07:56.130299 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:07:56.130283    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162\": container with ID starting with 4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162 not found: ID does not exist" containerID="4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"
Sep 05 20:07:56.130343 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.130303    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162"} err="failed to get container status \"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162\": rpc error: code = NotFound desc = could not find container \"4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162\": container with ID starting with 4ab3e34ff18e887a4b1e7fc1b28a6f747eec276fc74113ce5abca3ff01566162 not found: ID does not exist"
Sep 05 20:07:56.135690 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.135663    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:07:56.145828 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:56.145800    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ada83762-openshift-default-fb85f5dbd-kn8v6"]
Sep 05 20:07:57.043165 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:07:57.043133    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" path="/var/lib/kubelet/pods/3a7e80d9-c9e0-4ec0-ac85-4fba08533d66/volumes"
Sep 05 20:08:00.498934 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:00.498894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:00.561037 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:00.561005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:01.172262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:01.172232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:02.734413 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:02.734374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:07.591693 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:07.591657    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:07.699648 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:07.699611    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:07.796895 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:07.796861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:11.057680 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:11.057651    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:08:11.061063 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:11.061041    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:08:11.195459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:11.195427    2575 scope.go:117] "RemoveContainer" containerID="457cc5efc727bd8510d7aff4797250f36a160824e1e99968d1ea2f1f8d0002ea"
Sep 05 20:08:14.628224 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:14.628169    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:08:18.315139 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:18.315098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:18.538847 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:18.538818    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:31.540135 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:31.540099    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:08:39.433641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:39.433554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:43.532994 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:43.532941    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:08:45.466139 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:45.466100    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:45.526696 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:45.526648    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:45.799196 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:45.799089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:46.070107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070072    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:08:46.070733 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070713    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.070733 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070732    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.070904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070756    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.070904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070765    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.070904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070877    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.070904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.070891    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7e80d9-c9e0-4ec0-ac85-4fba08533d66" containerName="istio-proxy"
Sep 05 20:08:46.074514 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.074492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.076930 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.076906    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1aa465-openshift-default-dockercfg-qgzfb\""
Sep 05 20:08:46.084548 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.084527    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:08:46.096963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.096940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:46.210395 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.210337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:46.233492 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233638 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233638 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233638 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hbj\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.233987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.233838    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.334963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.334871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.334963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.334913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hbj\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335699 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.335786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.335764    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.336137 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.336114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.337375 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.337349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.337684 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.337668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.343554 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.343530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.343719 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.343704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hbj\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj\") pod \"e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") " pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.386516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.386494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:46.663581 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.663558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:08:46.665211 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:08:46.665165    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc4fc7d_908b_4f31_aa86_2e4b250859d0.slice/crio-08ddbee817887a07af366cccc02d00fdb447ed7c4fd92d860add18fcb61eebd4 WatchSource:0}: Error finding container 08ddbee817887a07af366cccc02d00fdb447ed7c4fd92d860add18fcb61eebd4: Status 404 returned error can't find the container with id 08ddbee817887a07af366cccc02d00fdb447ed7c4fd92d860add18fcb61eebd4
Sep 05 20:08:46.667802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.667769    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:08:46.667875 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.667835    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:08:46.667875 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:46.667862    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:08:47.367901 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:47.367863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerStarted","Data":"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"}
Sep 05 20:08:47.368080 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:47.367931    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerStarted","Data":"08ddbee817887a07af366cccc02d00fdb447ed7c4fd92d860add18fcb61eebd4"}
Sep 05 20:08:47.387136 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:47.387109    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:47.391143 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:47.391089    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podStartSLOduration=1.39107296 podStartE2EDuration="1.39107296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:47.389695592 +0000 UTC m=+1536.806769018" watchObservedRunningTime="2026-09-05 20:08:47.39107296 +0000 UTC m=+1536.808146377"
Sep 05 20:08:47.393239 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:47.393216    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:48.372965 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:48.372934    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:48.374031 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:48.374012    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:08:49.270436 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:49.270404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:49.368986 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:49.368944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:50.466403 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:50.466367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:50.566100 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:50.566065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:50.950894 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:50.950862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:51.051963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:51.051927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:51.379925 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:51.379886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:54.460273 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:54.460239    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:55.437655 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:55.437610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:55.658943 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:55.658909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:08:58.256983 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:08:58.256941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:02.246086 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:02.246051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:09:02.249056 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:02.249026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:02.351248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:02.351210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:11.218486 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:11.218450    2575 scope.go:117] "RemoveContainer" containerID="8c53e63acbdd531ec054715df784acd84634ef659e2a936af16dad4d189a6a96"
Sep 05 20:09:11.502615 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:11.502528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:13.513577 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.513543    2575 generic.go:356] "Generic (PLEG): container finished" podID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerID="b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f" exitCode=0
Sep 05 20:09:13.513577 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.513579    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerDied","Data":"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"}
Sep 05 20:09:13.514083 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.513984    2575 scope.go:117] "RemoveContainer" containerID="b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"
Sep 05 20:09:13.515155 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.515124    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:09:13.515225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.515213    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:09:13.515263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:13.515244    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:09:14.240099 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:14.240065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:14.520506 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:14.520412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerStarted","Data":"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"}
Sep 05 20:09:14.682166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:14.682131    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:15.004243 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:15.004173    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:15.337150 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:15.337111    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:15.387356 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:15.387320    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:09:15.389034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:15.388987    2575 patch_prober.go:28] interesting pod/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 05 20:09:15.389338 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:15.389290    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 05 20:09:16.387659 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:16.387609    2575 patch_prober.go:28] interesting pod/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 05 20:09:16.388172 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:16.387700    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 05 20:09:17.390114 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:17.390071    2575 patch_prober.go:28] interesting pod/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 05 20:09:17.390712 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:17.390682    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 05 20:09:18.219895 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:18.219858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:18.321135 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:18.321097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:18.373330 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:18.373295    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:09:18.387903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:18.387867    2575 patch_prober.go:28] interesting pod/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 05 20:09:18.388247 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:18.388215    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 05 20:09:19.364347 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.364301    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:09:19.366889 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.366865    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.369581 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.369555    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dbbedd-serving-cert\""
Sep 05 20:09:19.393591 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.390611    2575 patch_prober.go:28] interesting pod/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 05 20:09:19.393591 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.390702    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 05 20:09:19.402146 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.402111    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:09:19.406776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.406736    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:19.459106 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.459067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x788z\" (UniqueName: \"kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.459323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.459128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.459323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.459234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.459323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.459288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.506450 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.506405    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:19.560705 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.560667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.560898 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.560723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.560898 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.560788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x788z\" (UniqueName: \"kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.560898 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.560832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.561075 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:09:19.560939    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-serving-cert: secret "maas-api-e2e-api-b-dbbedd-serving-cert" not found
Sep 05 20:09:19.561075 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:09:19.561013    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls podName:5333ab26-cd77-449a-a6bf-2e5202719c54 nodeName:}" failed. No retries permitted until 2026-09-05 20:09:20.060993685 +0000 UTC m=+1569.478067100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls") pod "maas-api-e2e-api-b-dbbedd-776677877c-92xh4" (UID: "5333ab26-cd77-449a-a6bf-2e5202719c54") : secret "maas-api-e2e-api-b-dbbedd-serving-cert" not found
Sep 05 20:09:19.561504 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.561478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.563289 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.563263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.570301 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.570254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x788z\" (UniqueName: \"kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:19.628624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.628542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:19.996819 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:19.996734    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:20.065608 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.065573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:20.067966 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.067940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") pod \"maas-api-e2e-api-b-dbbedd-776677877c-92xh4\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:20.098317 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.098281    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:20.287887 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.287793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:20.405429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.405398    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:09:20.406526 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.406502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:09:20.563423 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.563385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:09:20.565051 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:09:20.565023    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5333ab26_cd77_449a_a6bf_2e5202719c54.slice/crio-5890a2b7b5af9d0a1188e508f3d1943ee3f4115ae7ef42bf3d7589e0a41746e3 WatchSource:0}: Error finding container 5890a2b7b5af9d0a1188e508f3d1943ee3f4115ae7ef42bf3d7589e0a41746e3: Status 404 returned error can't find the container with id 5890a2b7b5af9d0a1188e508f3d1943ee3f4115ae7ef42bf3d7589e0a41746e3
Sep 05 20:09:20.567053 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:20.567035    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 20:09:21.559689 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:21.559653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" event={"ID":"5333ab26-cd77-449a-a6bf-2e5202719c54","Type":"ContainerStarted","Data":"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"}
Sep 05 20:09:21.560119 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:21.559703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" event={"ID":"5333ab26-cd77-449a-a6bf-2e5202719c54","Type":"ContainerStarted","Data":"5890a2b7b5af9d0a1188e508f3d1943ee3f4115ae7ef42bf3d7589e0a41746e3"}
Sep 05 20:09:21.560119 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:21.559784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:21.581457 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:21.581383    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" podStartSLOduration=2.245773693 podStartE2EDuration="2.581360505s" totalImagesPullingTime="335.586812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:09:21.578967471 +0000 UTC m=+1570.996040923" watchObservedRunningTime="2026-09-05 20:09:21.581360505 +0000 UTC m=+1570.998433930"
Sep 05 20:09:23.250411 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:23.250379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:27.570419 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:27.570386    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:09:29.226067 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:29.226033    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:09:40.394623 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:40.394580    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:43.180538 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:43.180497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:46.204177 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:46.204142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:46.881930 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:46.881896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:47.690309 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:47.690269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:53.777463 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:53.777428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:54.783949 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:54.783916    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:09:57.780234 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:09:57.780198    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:10:01.489682 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:01.489644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:01.555313 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:01.555273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:01.655686 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:01.655654    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:01.741785 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:01.741701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:02.029062 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:02.028982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:07.911767 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:07.911681    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:07.971166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:07.971127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:08.139942 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:08.139910    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:10:08.140238 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:08.140203    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy" containerID="cri-o://6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a" gracePeriod=30
Sep 05 20:10:08.161967 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:08.161891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:08.600961 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:08.600922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:11.242130 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:11.242098    2575 scope.go:117] "RemoveContainer" containerID="79f8d841ed172bbcf822743566ee6ec89ea2228231c2b87a2d4f3120f247a692"
Sep 05 20:10:11.250361 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:11.250344    2575 scope.go:117] "RemoveContainer" containerID="e212b4caa61ba0e72697e9924f1e61bff3e6107a9e5a400cc4b7b21a15c9eb12"
Sep 05 20:10:13.398358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.398324    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:10:13.493660 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493587    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493660 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493623    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493660 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493653    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493692    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493711    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493735    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493814    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5hbj\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token\") pod \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\" (UID: \"9bc4fc7d-908b-4f31-aa86-2e4b250859d0\") "
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493898    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:13.493923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.493912    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494082    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494096    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494313    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494333    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494347    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494361    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.494372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.494373    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.496040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.496012    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:13.496169 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.496066    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj" (OuterVolumeSpecName: "kube-api-access-d5hbj") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "kube-api-access-d5hbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:13.496169 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.496072    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:13.496169 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.496089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9bc4fc7d-908b-4f31-aa86-2e4b250859d0" (UID: "9bc4fc7d-908b-4f31-aa86-2e4b250859d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:10:13.595303 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.595267    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.595303 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.595295    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hbj\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-kube-api-access-d5hbj\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.595303 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.595305    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.595534 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.595314    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc4fc7d-908b-4f31-aa86-2e4b250859d0-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:13.827031 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.826993    2575 generic.go:356] "Generic (PLEG): container finished" podID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerID="6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a" exitCode=0
Sep 05 20:10:13.827226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.827035    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"
Sep 05 20:10:13.827226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.827072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerDied","Data":"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"}
Sep 05 20:10:13.827226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.827125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49" event={"ID":"9bc4fc7d-908b-4f31-aa86-2e4b250859d0","Type":"ContainerDied","Data":"08ddbee817887a07af366cccc02d00fdb447ed7c4fd92d860add18fcb61eebd4"}
Sep 05 20:10:13.827226 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.827146    2575 scope.go:117] "RemoveContainer" containerID="6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"
Sep 05 20:10:13.836735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.836719    2575 scope.go:117] "RemoveContainer" containerID="b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"
Sep 05 20:10:13.845758 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.845742    2575 scope.go:117] "RemoveContainer" containerID="6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"
Sep 05 20:10:13.845992 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:13.845975    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a\": container with ID starting with 6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a not found: ID does not exist" containerID="6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"
Sep 05 20:10:13.846040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.845999    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a"} err="failed to get container status \"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a\": rpc error: code = NotFound desc = could not find container \"6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a\": container with ID starting with 6c844a78525b7d7dec0a017cee1bdc2d2caf031e20bd9ba802bb664deb55279a not found: ID does not exist"
Sep 05 20:10:13.846040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.846015    2575 scope.go:117] "RemoveContainer" containerID="b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"
Sep 05 20:10:13.846253 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:13.846234    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f\": container with ID starting with b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f not found: ID does not exist" containerID="b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"
Sep 05 20:10:13.846305 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.846260    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f"} err="failed to get container status \"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f\": rpc error: code = NotFound desc = could not find container \"b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f\": container with ID starting with b152f2d57fbd5a34d7d26b4c571c8b2d016f9f529afed770239bbd44381f992f not found: ID does not exist"
Sep 05 20:10:13.851162 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.851139    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:10:13.855605 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:13.855585    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1aa465-openshift-default-798f8fb744-hmw49"]
Sep 05 20:10:14.691573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:14.691539    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:15.043673 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:15.043594    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" path="/var/lib/kubelet/pods/9bc4fc7d-908b-4f31-aa86-2e4b250859d0/volumes"
Sep 05 20:10:16.425607 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:16.425572    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:10:18.691352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:18.691315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.248114 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.248074    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:10:24.248614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.248308    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" podUID="5333ab26-cd77-449a-a6bf-2e5202719c54" containerName="maas-api" containerID="cri-o://eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d" gracePeriod=30
Sep 05 20:10:24.282465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.282435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.360993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.360959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.425243 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.425157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.492776 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.492750    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.503377 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.503322    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:10:24.559652 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.559605    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.607209 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607152    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca\") pod \"5333ab26-cd77-449a-a6bf-2e5202719c54\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") "
Sep 05 20:10:24.607386 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607349    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs\") pod \"5333ab26-cd77-449a-a6bf-2e5202719c54\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") "
Sep 05 20:10:24.607448 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607390    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") pod \"5333ab26-cd77-449a-a6bf-2e5202719c54\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") "
Sep 05 20:10:24.607448 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607434    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x788z\" (UniqueName: \"kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z\") pod \"5333ab26-cd77-449a-a6bf-2e5202719c54\" (UID: \"5333ab26-cd77-449a-a6bf-2e5202719c54\") "
Sep 05 20:10:24.607614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607582    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5333ab26-cd77-449a-a6bf-2e5202719c54" (UID: "5333ab26-cd77-449a-a6bf-2e5202719c54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:24.607786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.607765    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5333ab26-cd77-449a-a6bf-2e5202719c54-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:24.609513 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.609484    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5333ab26-cd77-449a-a6bf-2e5202719c54" (UID: "5333ab26-cd77-449a-a6bf-2e5202719c54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:10:24.609617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.609594    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5333ab26-cd77-449a-a6bf-2e5202719c54" (UID: "5333ab26-cd77-449a-a6bf-2e5202719c54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:10:24.609660 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.609605    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z" (OuterVolumeSpecName: "kube-api-access-x788z") pod "5333ab26-cd77-449a-a6bf-2e5202719c54" (UID: "5333ab26-cd77-449a-a6bf-2e5202719c54"). InnerVolumeSpecName "kube-api-access-x788z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:24.709008 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.708965    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:24.709008 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.708997    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5333ab26-cd77-449a-a6bf-2e5202719c54-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:24.709008 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.709006    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x788z\" (UniqueName: \"kubernetes.io/projected/5333ab26-cd77-449a-a6bf-2e5202719c54-kube-api-access-x788z\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:24.836626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.836587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:24.884839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.884803    2575 generic.go:356] "Generic (PLEG): container finished" podID="5333ab26-cd77-449a-a6bf-2e5202719c54" containerID="eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d" exitCode=0
Sep 05 20:10:24.885038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.884848    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"
Sep 05 20:10:24.885038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.884894    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" event={"ID":"5333ab26-cd77-449a-a6bf-2e5202719c54","Type":"ContainerDied","Data":"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"}
Sep 05 20:10:24.885038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.884981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4" event={"ID":"5333ab26-cd77-449a-a6bf-2e5202719c54","Type":"ContainerDied","Data":"5890a2b7b5af9d0a1188e508f3d1943ee3f4115ae7ef42bf3d7589e0a41746e3"}
Sep 05 20:10:24.885038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.885013    2575 scope.go:117] "RemoveContainer" containerID="eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"
Sep 05 20:10:24.894669 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.894650    2575 scope.go:117] "RemoveContainer" containerID="eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"
Sep 05 20:10:24.894946 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:24.894927    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d\": container with ID starting with eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d not found: ID does not exist" containerID="eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"
Sep 05 20:10:24.894989 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.894954    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d"} err="failed to get container status \"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d\": rpc error: code = NotFound desc = could not find container \"eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d\": container with ID starting with eb94e6cea79302b9837525fa42a017245355cdf217dbb27bf279d04548ed093d not found: ID does not exist"
Sep 05 20:10:24.906667 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.906646    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:10:24.910414 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:24.910391    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dbbedd-776677877c-92xh4"]
Sep 05 20:10:25.042695 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:25.042659    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5333ab26-cd77-449a-a6bf-2e5202719c54" path="/var/lib/kubelet/pods/5333ab26-cd77-449a-a6bf-2e5202719c54/volumes"
Sep 05 20:10:29.529607 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:29.529568    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:10:31.275111 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:31.275077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:32.414671 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:32.414638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:35.243172 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:35.243136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:36.232560 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.232523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:36.652549 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.652516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:36.948213 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948107    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:10:36.948762 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948736    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5333ab26-cd77-449a-a6bf-2e5202719c54" containerName="maas-api"
Sep 05 20:10:36.948762 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948761    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5333ab26-cd77-449a-a6bf-2e5202719c54" containerName="maas-api"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948787    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948796    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948815    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948821    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948916    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948928    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc4fc7d-908b-4f31-aa86-2e4b250859d0" containerName="istio-proxy"
Sep 05 20:10:36.948971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.948942    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5333ab26-cd77-449a-a6bf-2e5202719c54" containerName="maas-api"
Sep 05 20:10:36.952095 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.952073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:36.954828 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.954806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d0f6b5-openshift-default-dockercfg-dkjpx\""
Sep 05 20:10:36.966779 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.966752    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:10:36.977009 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:36.976964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:37.137975 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.137932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.137989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.138425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.138375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cwt\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239659 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239659 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239659 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cwt\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.239946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.239831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.240310 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.240071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.240310 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.240246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.240310 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.240262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.240310 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.240289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.240641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.240620    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.242103 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.242076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.242309 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.242289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.246614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.246579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:37.249319 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.249294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.249425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.249410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cwt\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt\") pod \"e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") " pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.262284 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.262266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:37.541839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.541813    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:10:37.543562 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:10:37.543537    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0089cdc3_4e30_46d3_ab7e_e5dcbac2cec8.slice/crio-81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca WatchSource:0}: Error finding container 81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca: Status 404 returned error can't find the container with id 81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca
Sep 05 20:10:37.546174 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.546138    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:37.546296 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.546245    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:37.546367 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.546292    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:37.959308 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.959271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerStarted","Data":"07910360db609348b6db00860fcd317fae58fff09560f409a5fb24a2346a55bb"}
Sep 05 20:10:37.959694 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.959325    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerStarted","Data":"81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca"}
Sep 05 20:10:37.982729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:37.982676    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podStartSLOduration=1.9826610260000002 podStartE2EDuration="1.982661026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:37.980990997 +0000 UTC m=+1647.398064436" watchObservedRunningTime="2026-09-05 20:10:37.982661026 +0000 UTC m=+1647.399734497"
Sep 05 20:10:38.263320 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:38.263240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:38.269657 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:38.269628    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:38.964141 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:38.964103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:38.965044 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:38.965024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:39.849853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:39.849820    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:39.950634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:39.950602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.103320 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.103280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.199171 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.199141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.254644 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.254618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.589657 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.589628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.663902 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.663867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:41.743865 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:41.743830    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:10:42.342608 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.342566    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:10:42.343128 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.342843    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" podUID="ec84bc48-c167-4002-b8c0-49da6533818f" containerName="maas-api" containerID="cri-o://dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1" gracePeriod=30
Sep 05 20:10:42.372852 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.372822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.442332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.442265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.502359 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.502323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.587626 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.587597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.596378 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.596313    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:10:42.645921 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.645891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.694484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.694457    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs\") pod \"ec84bc48-c167-4002-b8c0-49da6533818f\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") "
Sep 05 20:10:42.694634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.694514    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca\") pod \"ec84bc48-c167-4002-b8c0-49da6533818f\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") "
Sep 05 20:10:42.694634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.694545    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rrz\" (UniqueName: \"kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz\") pod \"ec84bc48-c167-4002-b8c0-49da6533818f\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") "
Sep 05 20:10:42.694767 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.694647    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") pod \"ec84bc48-c167-4002-b8c0-49da6533818f\" (UID: \"ec84bc48-c167-4002-b8c0-49da6533818f\") "
Sep 05 20:10:42.694909 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.694875    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec84bc48-c167-4002-b8c0-49da6533818f" (UID: "ec84bc48-c167-4002-b8c0-49da6533818f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:42.695132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.695112    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec84bc48-c167-4002-b8c0-49da6533818f-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.696717 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.696696    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec84bc48-c167-4002-b8c0-49da6533818f" (UID: "ec84bc48-c167-4002-b8c0-49da6533818f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:10:42.697095 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.697075    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz" (OuterVolumeSpecName: "kube-api-access-g8rrz") pod "ec84bc48-c167-4002-b8c0-49da6533818f" (UID: "ec84bc48-c167-4002-b8c0-49da6533818f"). InnerVolumeSpecName "kube-api-access-g8rrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:42.697152 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.697113    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec84bc48-c167-4002-b8c0-49da6533818f" (UID: "ec84bc48-c167-4002-b8c0-49da6533818f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:10:42.795765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.795727    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rrz\" (UniqueName: \"kubernetes.io/projected/ec84bc48-c167-4002-b8c0-49da6533818f-kube-api-access-g8rrz\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.795765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.795757    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.795765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.795766    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec84bc48-c167-4002-b8c0-49da6533818f-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.946388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.946303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:42.985508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.985471    2575 generic.go:356] "Generic (PLEG): container finished" podID="ec84bc48-c167-4002-b8c0-49da6533818f" containerID="dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1" exitCode=0
Sep 05 20:10:42.985687 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.985529    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"
Sep 05 20:10:42.985745 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.985550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" event={"ID":"ec84bc48-c167-4002-b8c0-49da6533818f","Type":"ContainerDied","Data":"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"}
Sep 05 20:10:42.985790 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.985775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm" event={"ID":"ec84bc48-c167-4002-b8c0-49da6533818f","Type":"ContainerDied","Data":"1ac0889f746499d9d95e642a83b8629dcdb22a1934d54e50a7e625989d4882f1"}
Sep 05 20:10:42.985827 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.985803    2575 scope.go:117] "RemoveContainer" containerID="dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"
Sep 05 20:10:42.995282 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.995261    2575 scope.go:117] "RemoveContainer" containerID="dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"
Sep 05 20:10:42.995577 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:42.995556    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1\": container with ID starting with dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1 not found: ID does not exist" containerID="dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"
Sep 05 20:10:42.995627 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:42.995586    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1"} err="failed to get container status \"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1\": rpc error: code = NotFound desc = could not find container \"dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1\": container with ID starting with dccfc8e49b27e1c1b96938a97bfe2fd0340b851bf7ca006a058b3baebe5ac9f1 not found: ID does not exist"
Sep 05 20:10:43.009132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:43.009105    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:10:43.014923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:43.014902    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1436b9-5f886889d7-zkzzm"]
Sep 05 20:10:43.043970 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:43.043939    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec84bc48-c167-4002-b8c0-49da6533818f" path="/var/lib/kubelet/pods/ec84bc48-c167-4002-b8c0-49da6533818f/volumes"
Sep 05 20:10:48.911735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:48.911701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:49.010548 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:49.010505    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:49.143691 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:49.143661    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:10:49.143899 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:49.143874    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy" containerID="cri-o://0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751" gracePeriod=30
Sep 05 20:10:49.160479 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:49.160451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:49.583064 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:49.583025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:52.725419 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:52.725380    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:10:52.901527 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:52.901486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:10:54.400256 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.400220    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:10:54.515789 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515695    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.515789 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515756    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515854    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515961    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516030 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.515991    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516063    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516103    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mskxs\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs\") pod \"e5083df0-988b-497c-8866-c4bf1ce77d94\" (UID: \"e5083df0-988b-497c-8866-c4bf1ce77d94\") "
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516105    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516209    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516108    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516439    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516510    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516526    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516536    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516545    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5083df0-988b-497c-8866-c4bf1ce77d94-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.516685 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.516554    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.518338 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.518314    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs" (OuterVolumeSpecName: "kube-api-access-mskxs") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "kube-api-access-mskxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:54.518501 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.518474    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:54.518621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.518596    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:10:54.518679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.518605    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5083df0-988b-497c-8866-c4bf1ce77d94" (UID: "e5083df0-988b-497c-8866-c4bf1ce77d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:54.617352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.617314    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.617352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.617346    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.617352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.617355    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mskxs\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-kube-api-access-mskxs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:54.617581 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:54.617364    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5083df0-988b-497c-8866-c4bf1ce77d94-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:10:55.056867 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.056840    2575 generic.go:356] "Generic (PLEG): container finished" podID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerID="0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751" exitCode=0
Sep 05 20:10:55.056987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.056881    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"
Sep 05 20:10:55.056987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.056884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerDied","Data":"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"}
Sep 05 20:10:55.056987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.056924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd" event={"ID":"e5083df0-988b-497c-8866-c4bf1ce77d94","Type":"ContainerDied","Data":"b3d9a3c6c61746a02b5c0e0c183f975c3f05eb88707341882fcbc556ed6c9f7d"}
Sep 05 20:10:55.056987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.056945    2575 scope.go:117] "RemoveContainer" containerID="0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"
Sep 05 20:10:55.066323 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.066306    2575 scope.go:117] "RemoveContainer" containerID="6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"
Sep 05 20:10:55.074137 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.074118    2575 scope.go:117] "RemoveContainer" containerID="0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"
Sep 05 20:10:55.074386 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:55.074360    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751\": container with ID starting with 0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751 not found: ID does not exist" containerID="0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"
Sep 05 20:10:55.074491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.074397    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751"} err="failed to get container status \"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751\": rpc error: code = NotFound desc = could not find container \"0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751\": container with ID starting with 0792e6a5b6a8ce963324065647749f553936ecace5fcda50dffef6440eb22751 not found: ID does not exist"
Sep 05 20:10:55.074491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.074419    2575 scope.go:117] "RemoveContainer" containerID="6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"
Sep 05 20:10:55.074658 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:10:55.074640    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d\": container with ID starting with 6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d not found: ID does not exist" containerID="6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"
Sep 05 20:10:55.074700 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.074662    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d"} err="failed to get container status \"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d\": rpc error: code = NotFound desc = could not find container \"6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d\": container with ID starting with 6d2de215a2cef751fffb84c5238c3825aacb0b19234558b59bb46c051bc11c6d not found: ID does not exist"
Sep 05 20:10:55.079053 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.079030    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:10:55.083210 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:55.083172    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1436b9-openshift-default-86f5d54b48-vqbfd"]
Sep 05 20:10:57.043824 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:57.043786    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" path="/var/lib/kubelet/pods/e5083df0-988b-497c-8866-c4bf1ce77d94/volumes"
Sep 05 20:10:58.075907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.075878    2575 generic.go:356] "Generic (PLEG): container finished" podID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerID="07910360db609348b6db00860fcd317fae58fff09560f409a5fb24a2346a55bb" exitCode=0
Sep 05 20:10:58.076322 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.075960    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerDied","Data":"07910360db609348b6db00860fcd317fae58fff09560f409a5fb24a2346a55bb"}
Sep 05 20:10:58.076440 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.076423    2575 scope.go:117] "RemoveContainer" containerID="07910360db609348b6db00860fcd317fae58fff09560f409a5fb24a2346a55bb"
Sep 05 20:10:58.077535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.077494    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:58.077640 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.077586    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:58.077704 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:58.077648    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:10:59.086280 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:59.086222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerStarted","Data":"07abd24c074b4133a9e82e85ea3e81574f191bdb392a8b44ee5804002e99af8d"}
Sep 05 20:10:59.263125 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:59.263085    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:10:59.263368 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:59.263345    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 05 20:10:59.263445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:59.263405    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 05 20:10:59.605470 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:10:59.605435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:00.144109 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.144000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:11:00.144602 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.144285    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" podUID="33ec1803-6b7f-472a-b027-a5b132efdf61" containerName="maas-api" containerID="cri-o://8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62" gracePeriod=30
Sep 05 20:11:00.171113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.171060    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:00.246249 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.246214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:00.263196 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.263160    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 05 20:11:00.263299 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.263226    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 05 20:11:00.309603 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.309567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:00.398974 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.398751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:00.425746 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.425720    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.493500    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbt6l\" (UniqueName: \"kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l\") pod \"33ec1803-6b7f-472a-b027-a5b132efdf61\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") "
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.493562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs\") pod \"33ec1803-6b7f-472a-b027-a5b132efdf61\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") "
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.493650    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") pod \"33ec1803-6b7f-472a-b027-a5b132efdf61\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") "
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.493753    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca\") pod \"33ec1803-6b7f-472a-b027-a5b132efdf61\" (UID: \"33ec1803-6b7f-472a-b027-a5b132efdf61\") "
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.494508    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33ec1803-6b7f-472a-b027-a5b132efdf61" (UID: "33ec1803-6b7f-472a-b027-a5b132efdf61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:11:00.500848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.500649    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33ec1803-6b7f-472a-b027-a5b132efdf61" (UID: "33ec1803-6b7f-472a-b027-a5b132efdf61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:11:00.502601 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.502544    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l" (OuterVolumeSpecName: "kube-api-access-cbt6l") pod "33ec1803-6b7f-472a-b027-a5b132efdf61" (UID: "33ec1803-6b7f-472a-b027-a5b132efdf61"). InnerVolumeSpecName "kube-api-access-cbt6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:11:00.504269 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.504217    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33ec1803-6b7f-472a-b027-a5b132efdf61" (UID: "33ec1803-6b7f-472a-b027-a5b132efdf61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:11:00.595210 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.594840    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:00.595210 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.594900    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ec1803-6b7f-472a-b027-a5b132efdf61-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:00.595210 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.594922    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbt6l\" (UniqueName: \"kubernetes.io/projected/33ec1803-6b7f-472a-b027-a5b132efdf61-kube-api-access-cbt6l\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:00.595210 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.594936    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ec1803-6b7f-472a-b027-a5b132efdf61-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:00.724735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:00.724611    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:01.102107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.102069    2575 generic.go:356] "Generic (PLEG): container finished" podID="33ec1803-6b7f-472a-b027-a5b132efdf61" containerID="8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62" exitCode=0
Sep 05 20:11:01.102334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.102122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" event={"ID":"33ec1803-6b7f-472a-b027-a5b132efdf61","Type":"ContainerDied","Data":"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"}
Sep 05 20:11:01.102334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.102167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx" event={"ID":"33ec1803-6b7f-472a-b027-a5b132efdf61","Type":"ContainerDied","Data":"946ce8ba8bc78029afca9475d4ce505a34229202abd64fcb0db34d3921bb3dae"}
Sep 05 20:11:01.102334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.102126    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"
Sep 05 20:11:01.102334 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.102215    2575 scope.go:117] "RemoveContainer" containerID="8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"
Sep 05 20:11:01.111387 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.111362    2575 scope.go:117] "RemoveContainer" containerID="8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"
Sep 05 20:11:01.111677 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:11:01.111658    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62\": container with ID starting with 8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62 not found: ID does not exist" containerID="8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"
Sep 05 20:11:01.111735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.111688    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62"} err="failed to get container status \"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62\": rpc error: code = NotFound desc = could not find container \"8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62\": container with ID starting with 8efc8af116c8698649cc5d3463f6565358c890106ab7a664c67c6b2a33e6bd62 not found: ID does not exist"
Sep 05 20:11:01.119809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.119784    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:11:01.124031 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.124010    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca4fba-77bdc46774-4dlcx"]
Sep 05 20:11:01.263064 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.263028    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 05 20:11:01.263480 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:01.263094    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 05 20:11:02.099580 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:02.099535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:02.262940 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:02.262904    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 05 20:11:02.263106 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:02.262962    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 05 20:11:03.043428 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:03.043390    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ec1803-6b7f-472a-b027-a5b132efdf61" path="/var/lib/kubelet/pods/33ec1803-6b7f-472a-b027-a5b132efdf61/volumes"
Sep 05 20:11:03.262741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:03.262703    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 05 20:11:03.262912 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:03.262762    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 05 20:11:03.343771 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:03.343735    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:11:04.338128 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:04.338094    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:11:04.338509 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:04.338487    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:11:04.339489 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:04.339461    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:11:04.793046 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:04.792962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:05.444307 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.444273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:05.728952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.728876    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:11:05.729313 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729297    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:05.729372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729314    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:05.729372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729332    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ec1803-6b7f-472a-b027-a5b132efdf61" containerName="maas-api"
Sep 05 20:11:05.729372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729338    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ec1803-6b7f-472a-b027-a5b132efdf61" containerName="maas-api"
Sep 05 20:11:05.729372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729349    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec84bc48-c167-4002-b8c0-49da6533818f" containerName="maas-api"
Sep 05 20:11:05.729372 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729355    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec84bc48-c167-4002-b8c0-49da6533818f" containerName="maas-api"
Sep 05 20:11:05.729542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729420    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec84bc48-c167-4002-b8c0-49da6533818f" containerName="maas-api"
Sep 05 20:11:05.729542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729430    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:05.729542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729440    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:05.729542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.729450    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ec1803-6b7f-472a-b027-a5b132efdf61" containerName="maas-api"
Sep 05 20:11:05.732489 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.732468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.735496 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.735475    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6693ae-openshift-default-dockercfg-f72zc\""
Sep 05 20:11:05.745723 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.745698    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:11:05.755647 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.755621    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:05.847780 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.847743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.848358 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.848322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.848538 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.848519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.848674 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.848659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.848804 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.848790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.848945 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.848932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.849056 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.849043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.849162 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.849149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.849286 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.849271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.866984 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.866959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:05.949965 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.949932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.949990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950435 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950615 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950616    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950737 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950717    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.950785 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.950735    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.951080 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.951055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.952771 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.952746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.953060 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.953043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.959508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.959483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:05.959618 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:05.959602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs\") pod \"e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") " pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:06.042935 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:06.042864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:06.316617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:06.316565    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:11:06.319029 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:11:06.319002    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729835be_311b_4a09_a4c9_360c23e18856.slice/crio-02833d31df9ad5d64c14e3a4c886b83c3b861984564369ae0910c73eb238cb34 WatchSource:0}: Error finding container 02833d31df9ad5d64c14e3a4c886b83c3b861984564369ae0910c73eb238cb34: Status 404 returned error can't find the container with id 02833d31df9ad5d64c14e3a4c886b83c3b861984564369ae0910c73eb238cb34
Sep 05 20:11:06.321619 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:06.321584    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:06.321697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:06.321654    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:06.321697 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:06.321680    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:07.135961 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.135928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerStarted","Data":"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"}
Sep 05 20:11:07.136641 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.135985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerStarted","Data":"02833d31df9ad5d64c14e3a4c886b83c3b861984564369ae0910c73eb238cb34"}
Sep 05 20:11:07.158388 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.158323    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podStartSLOduration=2.15830511 podStartE2EDuration="2.15830511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:11:07.156927665 +0000 UTC m=+1676.574001101" watchObservedRunningTime="2026-09-05 20:11:07.15830511 +0000 UTC m=+1676.575378532"
Sep 05 20:11:07.312485 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.312444    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:07.413864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.413769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:07.547531 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.547499    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:11:07.547732 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.547709    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy" containerID="cri-o://cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e" gracePeriod=30
Sep 05 20:11:07.565637 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:07.565593    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:08.043507 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:08.043477    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:08.048429 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:08.048404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:08.141912 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:08.141882    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:08.142892 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:08.142871    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:09.039984 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:09.039953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:09.133425 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:09.133396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:10.309394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:10.309362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:10.369280 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:10.369244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:10.784440 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:10.784346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:10.877954 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:10.877918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:12.793519 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.793492    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:11:12.820848 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.820826    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.820963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.820861    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.820963 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.820933    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.820969    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821003    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd2kv\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821037    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821316 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821064    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821316 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data\") pod \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\" (UID: \"3c18c5bf-2369-4cb2-9dae-956abc0a82b0\") "
Sep 05 20:11:12.821316 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821234    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:11:12.821466 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821316    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:11:12.821466 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821446    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:11:12.821567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821502    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.821567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821521    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.821567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821525    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:11:12.821904 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.821860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:11:12.823176 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.823153    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:11:12.823176 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.823163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:11:12.823344 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.823265    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:11:12.823344 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.823329    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv" (OuterVolumeSpecName: "kube-api-access-kd2kv") pod "3c18c5bf-2369-4cb2-9dae-956abc0a82b0" (UID: "3c18c5bf-2369-4cb2-9dae-956abc0a82b0"). InnerVolumeSpecName "kube-api-access-kd2kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:11:12.922839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922805    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.922839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922834    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2kv\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-kube-api-access-kd2kv\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.922839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922844    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.923090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922853    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.923090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922861    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.923090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922870    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:12.923090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:12.922877    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c18c5bf-2369-4cb2-9dae-956abc0a82b0-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:11:13.166954 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.166868    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerID="cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e" exitCode=0
Sep 05 20:11:13.166954 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.166917    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"
Sep 05 20:11:13.166954 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.166944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerDied","Data":"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"}
Sep 05 20:11:13.167179 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.166995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd" event={"ID":"3c18c5bf-2369-4cb2-9dae-956abc0a82b0","Type":"ContainerDied","Data":"4ab3c5cf4e113d5d461bdae88e354f9c6d549fb5dbfbaa8fc3efe6f2187dee8a"}
Sep 05 20:11:13.167179 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.167017    2575 scope.go:117] "RemoveContainer" containerID="cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"
Sep 05 20:11:13.176054 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.176038    2575 scope.go:117] "RemoveContainer" containerID="12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"
Sep 05 20:11:13.184620 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.184599    2575 scope.go:117] "RemoveContainer" containerID="cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"
Sep 05 20:11:13.184942 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:11:13.184918    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e\": container with ID starting with cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e not found: ID does not exist" containerID="cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"
Sep 05 20:11:13.185018 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.184954    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e"} err="failed to get container status \"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e\": rpc error: code = NotFound desc = could not find container \"cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e\": container with ID starting with cc1594470f4ed3cece07433137f9cd4e88eb0951224c5a0baf1cbeb9b69c264e not found: ID does not exist"
Sep 05 20:11:13.185018 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.184978    2575 scope.go:117] "RemoveContainer" containerID="12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"
Sep 05 20:11:13.185308 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:11:13.185280    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5\": container with ID starting with 12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5 not found: ID does not exist" containerID="12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"
Sep 05 20:11:13.185430 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.185310    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5"} err="failed to get container status \"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5\": rpc error: code = NotFound desc = could not find container \"12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5\": container with ID starting with 12687d29d3eb5959ba14283dbd1960e756aa4efc064590b2e25bb4074739b6f5 not found: ID does not exist"
Sep 05 20:11:13.186040 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.186017    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:11:13.190111 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:13.190087    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca4fba-openshift-default-7dbd8b77fd-82sdd"]
Sep 05 20:11:14.632981 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:14.632944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:11:15.043290 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:15.043209    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" path="/var/lib/kubelet/pods/3c18c5bf-2369-4cb2-9dae-956abc0a82b0/volumes"
Sep 05 20:11:24.182363 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:24.182320    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 20:11:24.182785 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:24.182404    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:11:25.236356 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.236314    2575 generic.go:356] "Generic (PLEG): container finished" podID="729835be-311b-4a09-a4c9-360c23e18856" containerID="5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526" exitCode=0
Sep 05 20:11:25.236886 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.236392    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerDied","Data":"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"}
Sep 05 20:11:25.236950 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.236931    2575 scope.go:117] "RemoveContainer" containerID="5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"
Sep 05 20:11:25.238031 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.237998    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:25.238121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.238064    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:25.238121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:25.238096    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:11:26.246420 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:26.246380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerStarted","Data":"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"}
Sep 05 20:11:27.043952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:27.043924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:27.044150 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:27.044129    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 05 20:11:27.044215 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:27.044176    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 05 20:11:28.043748 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:28.043706    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 05 20:11:28.044250 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:28.043764    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 05 20:11:29.043824 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:29.043787    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 05 20:11:29.044349 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:29.043846    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 05 20:11:30.043483 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:30.043440    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 05 20:11:30.043662 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:30.043505    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 05 20:11:31.044120 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:31.044085    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 05 20:11:31.044619 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:31.044146    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 05 20:11:32.047567 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:32.047533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:32.048058 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:32.047921    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:32.048493 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:32.048473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:11:34.016445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:34.016351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:48.740622 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.740585    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:11:48.741000 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.740978    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:11:48.741000 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.740989    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:11:48.741071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.741008    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:11:48.741071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.741013    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:11:48.741071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.741024    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:48.741071 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.741029    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5083df0-988b-497c-8866-c4bf1ce77d94" containerName="istio-proxy"
Sep 05 20:11:48.741225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.741089    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:11:48.744070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.744054    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.747972 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.747950    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0f6b5\"/\"ipp-route-d0f6b5-kserve-self-signed-certs\""
Sep 05 20:11:48.749011 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.748993    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0f6b5\"/\"openshift-service-ca.crt\""
Sep 05 20:11:48.749011 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.749005    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0f6b5\"/\"kube-root-ca.crt\""
Sep 05 20:11:48.749154 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.749143    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0f6b5\"/\"default-dockercfg-4btsz\""
Sep 05 20:11:48.756794 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.756771    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:11:48.766631 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.766608    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:48.787487 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.787593 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.787644 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.787682 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.787682 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flmg\" (UniqueName: \"kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.787765 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.787714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888403 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888570 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888570 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flmg\" (UniqueName: \"kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888570 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888570 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888834 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888859    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.888930 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.888912    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.890839 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.890818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.891089 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.891074    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:48.896650 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:48.896624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flmg\" (UniqueName: \"kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg\") pod \"ipp-route-d0f6b5-kserve-799758fd55-tkkq9\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") " pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:49.053046 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:49.052960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:49.328082 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:49.328053    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:11:49.329777 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:11:49.329750    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45433a13_16ca_472b_8ff7_a79c2cd4991e.slice/crio-46afb88494e37f769ef4e0c43235643fb832e3f16eb594a78f1e5d68b197aa1e WatchSource:0}: Error finding container 46afb88494e37f769ef4e0c43235643fb832e3f16eb594a78f1e5d68b197aa1e: Status 404 returned error can't find the container with id 46afb88494e37f769ef4e0c43235643fb832e3f16eb594a78f1e5d68b197aa1e
Sep 05 20:11:49.355414 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:49.355388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" event={"ID":"45433a13-16ca-472b-8ff7-a79c2cd4991e","Type":"ContainerStarted","Data":"46afb88494e37f769ef4e0c43235643fb832e3f16eb594a78f1e5d68b197aa1e"}
Sep 05 20:11:50.361897 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:50.361866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" event={"ID":"45433a13-16ca-472b-8ff7-a79c2cd4991e","Type":"ContainerStarted","Data":"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"}
Sep 05 20:11:50.384482 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:50.384431    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" podStartSLOduration=2.384415034 podStartE2EDuration="2.384415034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:11:50.383955288 +0000 UTC m=+1719.801028711" watchObservedRunningTime="2026-09-05 20:11:50.384415034 +0000 UTC m=+1719.801488457"
Sep 05 20:11:51.898303 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:51.898267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:51.955679 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:51.955640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:11:52.731508 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:52.731472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:11:59.053399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:59.053363    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:59.053855 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:59.053422    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:59.065761 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:59.065738    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:11:59.422774 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:11:59.422744    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:12:11.430980 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:11.430947    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:12:15.156431 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.156395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:15.439447 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.439359    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:12:15.439903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.439883    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c18c5bf-2369-4cb2-9dae-956abc0a82b0" containerName="istio-proxy"
Sep 05 20:12:15.442924 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.442899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.445823 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.445803    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-14a23e-openshift-default-dockercfg-rw6x8\""
Sep 05 20:12:15.458091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.458067    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:12:15.467853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.467816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:15.546033 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.545991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.546587 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.546520    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxqj\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647322 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647322 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxqj\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647552 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647802 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647757    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647857 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647857 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.647952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.647868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.648121 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.648098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.648209 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.648151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.648209 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.648176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.649881 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.649856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.650009 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.649905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.656107 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.656088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.656242 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.656224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxqj\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj\") pod \"e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") " pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.753583 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.753501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:15.776996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:15.776955    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:16.030468 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.030440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:12:16.032695 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:12:16.032669    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8cd9ab_d4e1_4872_b862_e12376529b91.slice/crio-8dfcf05fb2d0a82644f7ab6cd872b6383b751bb72af0ac9f423739704ad2c5d1 WatchSource:0}: Error finding container 8dfcf05fb2d0a82644f7ab6cd872b6383b751bb72af0ac9f423739704ad2c5d1: Status 404 returned error can't find the container with id 8dfcf05fb2d0a82644f7ab6cd872b6383b751bb72af0ac9f423739704ad2c5d1
Sep 05 20:12:16.034935 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.034900    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:16.035026 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.034964    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:16.035026 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.034991    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:16.496135 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.496093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerStarted","Data":"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"}
Sep 05 20:12:16.496544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.496208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerStarted","Data":"8dfcf05fb2d0a82644f7ab6cd872b6383b751bb72af0ac9f423739704ad2c5d1"}
Sep 05 20:12:16.518650 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.518604    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podStartSLOduration=1.518591191 podStartE2EDuration="1.518591191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:12:16.517322347 +0000 UTC m=+1745.934395772" watchObservedRunningTime="2026-09-05 20:12:16.518591191 +0000 UTC m=+1745.935664614"
Sep 05 20:12:16.754718 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:16.754623    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:17.759314 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:17.759282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:18.414428 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:18.414388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:18.506538 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:18.506500    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:18.507632 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:18.507611    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:18.507968 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:18.507944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:19.665312 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.665277    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:19.669006 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.668982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.671743 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.671710    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-14a23e-serving-cert\""
Sep 05 20:12:19.672935 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.672911    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 20:12:19.673075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.673057    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:12:19.673156 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.673059    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 05 20:12:19.673156 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.673141    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 05 20:12:19.681514 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.681492    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:19.708399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.708346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:19.775914 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.775874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:19.787585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.786615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnq9\" (UniqueName: \"kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.787585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.786681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.787585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.786737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.787585 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.786795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.888442 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.888406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.888633 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.888471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.888633 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.888523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.888784 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.888640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnq9\" (UniqueName: \"kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.889225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.889172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.890992 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.890969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.891089 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.891016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.896726 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.896699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:19.896953 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.896935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnq9\" (UniqueName: \"kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9\") pod \"maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:19.983869 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:19.983780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:20.271969 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:20.271886    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:20.274783 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:12:20.274754    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd766c_b1f9_401d_8505_096d83b721e7.slice/crio-c0ec43d9d87da72f97da2a4756d968804d637d2faf2019ed5a0d88a9463e6025 WatchSource:0}: Error finding container c0ec43d9d87da72f97da2a4756d968804d637d2faf2019ed5a0d88a9463e6025: Status 404 returned error can't find the container with id c0ec43d9d87da72f97da2a4756d968804d637d2faf2019ed5a0d88a9463e6025
Sep 05 20:12:20.279894 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:20.279866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:20.368933 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:20.368898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:20.522471 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:20.522365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" event={"ID":"e6fd766c-b1f9-401d-8505-096d83b721e7","Type":"ContainerStarted","Data":"c0ec43d9d87da72f97da2a4756d968804d637d2faf2019ed5a0d88a9463e6025"}
Sep 05 20:12:21.534220 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:21.534161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" event={"ID":"e6fd766c-b1f9-401d-8505-096d83b721e7","Type":"ContainerStarted","Data":"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"}
Sep 05 20:12:21.534731 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:21.534335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:21.553100 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:21.553054    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" podStartSLOduration=2.212941108 podStartE2EDuration="2.553040531s" totalImagesPullingTime="340.099423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:12:21.552725871 +0000 UTC m=+1750.969799293" watchObservedRunningTime="2026-09-05 20:12:21.553040531 +0000 UTC m=+1750.970113953"
Sep 05 20:12:24.043864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:24.043826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:12:27.544173 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:27.544142    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:33.824061 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:33.823060    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:38016->10.133.0.61:15021: read: connection reset by peer" start-of-body=
Sep 05 20:12:33.824061 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:33.823143    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:38016->10.133.0.61:15021: read: connection reset by peer"
Sep 05 20:12:34.610407 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.610371    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerID="5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de" exitCode=0
Sep 05 20:12:34.610576 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.610434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerDied","Data":"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"}
Sep 05 20:12:34.610880 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.610855    2575 scope.go:117] "RemoveContainer" containerID="5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"
Sep 05 20:12:34.611937 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.611908    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:34.611989 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.611976    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:34.612025 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:34.612007    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 05 20:12:35.617823 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:35.617788    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerStarted","Data":"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"}
Sep 05 20:12:35.753844 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:35.753811    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:35.754023 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:35.754002    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 05 20:12:35.754076 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:35.754056    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 05 20:12:36.754093 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:36.754052    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 05 20:12:36.754524 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:36.754123    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 05 20:12:37.754291 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:37.754247    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 05 20:12:37.754686 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:37.754313    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 05 20:12:38.754777 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:38.754737    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 05 20:12:38.755177 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:38.754805    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 05 20:12:39.754664 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:39.754631    2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 05 20:12:39.754861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:39.754686    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 05 20:12:40.812208 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:40.812153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:40.813048 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:40.813021    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:40.814016 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:40.813991    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:12:43.404821 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:43.404786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.086112 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.086021    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:50.086579 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.086363    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" podUID="e6fd766c-b1f9-401d-8505-096d83b721e7" containerName="maas-api" containerID="cri-o://308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e" gracePeriod=30
Sep 05 20:12:50.117788 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.117751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.186944 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.186879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.251290 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.251254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.311998 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.311962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.335635 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.335612    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:50.408806 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.408722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs\") pod \"e6fd766c-b1f9-401d-8505-096d83b721e7\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") "
Sep 05 20:12:50.408960 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.408838    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls\") pod \"e6fd766c-b1f9-401d-8505-096d83b721e7\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") "
Sep 05 20:12:50.408960 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.408866    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca\") pod \"e6fd766c-b1f9-401d-8505-096d83b721e7\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") "
Sep 05 20:12:50.408960 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.408886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlnq9\" (UniqueName: \"kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9\") pod \"e6fd766c-b1f9-401d-8505-096d83b721e7\" (UID: \"e6fd766c-b1f9-401d-8505-096d83b721e7\") "
Sep 05 20:12:50.409296 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.409268    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6fd766c-b1f9-401d-8505-096d83b721e7" (UID: "e6fd766c-b1f9-401d-8505-096d83b721e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:12:50.410890 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.410859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6fd766c-b1f9-401d-8505-096d83b721e7" (UID: "e6fd766c-b1f9-401d-8505-096d83b721e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:12:50.411036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.410894    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6fd766c-b1f9-401d-8505-096d83b721e7" (UID: "e6fd766c-b1f9-401d-8505-096d83b721e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:12:50.411036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.410966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9" (OuterVolumeSpecName: "kube-api-access-rlnq9") pod "e6fd766c-b1f9-401d-8505-096d83b721e7" (UID: "e6fd766c-b1f9-401d-8505-096d83b721e7"). InnerVolumeSpecName "kube-api-access-rlnq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:12:50.509808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.509771    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:12:50.509808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.509804    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd766c-b1f9-401d-8505-096d83b721e7-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:12:50.509808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.509814    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnq9\" (UniqueName: \"kubernetes.io/projected/e6fd766c-b1f9-401d-8505-096d83b721e7-kube-api-access-rlnq9\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:12:50.510066 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.509823    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd766c-b1f9-401d-8505-096d83b721e7-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:12:50.669464 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.669372    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:50.698503 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.698466    2575 generic.go:356] "Generic (PLEG): container finished" podID="e6fd766c-b1f9-401d-8505-096d83b721e7" containerID="308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e" exitCode=0
Sep 05 20:12:50.698678 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.698510    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"
Sep 05 20:12:50.698678 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.698554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" event={"ID":"e6fd766c-b1f9-401d-8505-096d83b721e7","Type":"ContainerDied","Data":"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"}
Sep 05 20:12:50.698678 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.698610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf" event={"ID":"e6fd766c-b1f9-401d-8505-096d83b721e7","Type":"ContainerDied","Data":"c0ec43d9d87da72f97da2a4756d968804d637d2faf2019ed5a0d88a9463e6025"}
Sep 05 20:12:50.698678 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.698633    2575 scope.go:117] "RemoveContainer" containerID="308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"
Sep 05 20:12:50.708049 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.708032    2575 scope.go:117] "RemoveContainer" containerID="308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"
Sep 05 20:12:50.708362 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:12:50.708340    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e\": container with ID starting with 308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e not found: ID does not exist" containerID="308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"
Sep 05 20:12:50.708447 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.708369    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e"} err="failed to get container status \"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e\": rpc error: code = NotFound desc = could not find container \"308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e\": container with ID starting with 308b0f10f7f26c9fd75aadc663ffad42189b57f00cb995a7136a371b66c94d1e not found: ID does not exist"
Sep 05 20:12:50.724680 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.724636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:50.729304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:50.729267    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-14a23e-7c47bcd695-dl5mf"]
Sep 05 20:12:51.053907 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:51.053822    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fd766c-b1f9-401d-8505-096d83b721e7" path="/var/lib/kubelet/pods/e6fd766c-b1f9-401d-8505-096d83b721e7/volumes"
Sep 05 20:12:55.428013 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:55.427974    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:12:56.550133 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:56.550097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:56.609091 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:56.609061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:12:56.786346 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:56.786298    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:12:56.786666 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:56.786639    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy" containerID="cri-o://494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37" gracePeriod=30
Sep 05 20:12:56.808340 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:12:56.808306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:00.239198 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.239150    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"]
Sep 05 20:13:00.239630 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.239564    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fd766c-b1f9-401d-8505-096d83b721e7" containerName="maas-api"
Sep 05 20:13:00.239630 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.239578    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fd766c-b1f9-401d-8505-096d83b721e7" containerName="maas-api"
Sep 05 20:13:00.239718 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.239654    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fd766c-b1f9-401d-8505-096d83b721e7" containerName="maas-api"
Sep 05 20:13:00.242445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.242429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:00.245037 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.245020    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:13:00.245128 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.245062    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:13:00.250886 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.250861    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"]
Sep 05 20:13:00.312576 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.312544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7\" (UID: \"785b10c6-8470-4d5a-9cce-ed579a31f06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:00.414073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.414035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7\" (UID: \"785b10c6-8470-4d5a-9cce-ed579a31f06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:00.414694 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.414674    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7\" (UID: \"785b10c6-8470-4d5a-9cce-ed579a31f06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:00.478403 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.478364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:00.554816 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.554730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:00.824864 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:00.824835    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"]
Sep 05 20:13:00.827016 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:13:00.826991    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785b10c6_8470_4d5a_9cce_ed579a31f06a.slice/crio-ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a WatchSource:0}: Error finding container ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a: Status 404 returned error can't find the container with id ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a
Sep 05 20:13:01.756467 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:01.756426    2575 generic.go:356] "Generic (PLEG): container finished" podID="785b10c6-8470-4d5a-9cce-ed579a31f06a" containerID="766507f483f953791e727e66b4c026fb432c0b3f2cf34b53e7bab8cf598fc31a" exitCode=0
Sep 05 20:13:01.756888 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:01.756515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7" event={"ID":"785b10c6-8470-4d5a-9cce-ed579a31f06a","Type":"ContainerDied","Data":"766507f483f953791e727e66b4c026fb432c0b3f2cf34b53e7bab8cf598fc31a"}
Sep 05 20:13:01.756888 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:01.756564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7" event={"ID":"785b10c6-8470-4d5a-9cce-ed579a31f06a","Type":"ContainerStarted","Data":"ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a"}
Sep 05 20:13:02.036676 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.036654    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:13:02.133222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133173    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133233    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133262    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbxqj\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133392 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133317    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133398    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133436    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133573 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133522    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:02.133887 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133596    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert\") pod \"ac8cd9ab-d4e1-4872-b862-e12376529b91\" (UID: \"ac8cd9ab-d4e1-4872-b862-e12376529b91\") "
Sep 05 20:13:02.133887 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133714    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:02.133887 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133812    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:02.134032 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.133970    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:13:02.134032 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.134021    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.134132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.134042    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.134132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.134058    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.134132 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.134072    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.136022 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.135996    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:02.136022 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.135994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj" (OuterVolumeSpecName: "kube-api-access-nbxqj") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "kube-api-access-nbxqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:13:02.136171 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.136030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:13:02.136171 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.136042    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac8cd9ab-d4e1-4872-b862-e12376529b91" (UID: "ac8cd9ab-d4e1-4872-b862-e12376529b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:13:02.235456 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.235380    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.235456 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.235407    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.235456 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.235416    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8cd9ab-d4e1-4872-b862-e12376529b91-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.235456 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.235424    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxqj\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-kube-api-access-nbxqj\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.235456 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.235435    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8cd9ab-d4e1-4872-b862-e12376529b91-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:02.771012 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.770974    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerID="494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37" exitCode=0
Sep 05 20:13:02.771478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.771031    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"
Sep 05 20:13:02.771478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.771056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerDied","Data":"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"}
Sep 05 20:13:02.771478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.771129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph" event={"ID":"ac8cd9ab-d4e1-4872-b862-e12376529b91","Type":"ContainerDied","Data":"8dfcf05fb2d0a82644f7ab6cd872b6383b751bb72af0ac9f423739704ad2c5d1"}
Sep 05 20:13:02.771478 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.771152    2575 scope.go:117] "RemoveContainer" containerID="494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"
Sep 05 20:13:02.780634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.780617    2575 scope.go:117] "RemoveContainer" containerID="5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"
Sep 05 20:13:02.789041 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.789023    2575 scope.go:117] "RemoveContainer" containerID="494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"
Sep 05 20:13:02.789360 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:13:02.789335    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37\": container with ID starting with 494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37 not found: ID does not exist" containerID="494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"
Sep 05 20:13:02.789452 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.789365    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37"} err="failed to get container status \"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37\": rpc error: code = NotFound desc = could not find container \"494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37\": container with ID starting with 494003d49a78ac9bbb005aaf7b3bff476e12e42a2fe4c360fb118280d00b0d37 not found: ID does not exist"
Sep 05 20:13:02.789452 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.789385    2575 scope.go:117] "RemoveContainer" containerID="5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"
Sep 05 20:13:02.789685 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:13:02.789665    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de\": container with ID starting with 5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de not found: ID does not exist" containerID="5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"
Sep 05 20:13:02.789731 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.789701    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de"} err="failed to get container status \"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de\": rpc error: code = NotFound desc = could not find container \"5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de\": container with ID starting with 5c4955f37af16cb0d33dfd17321b30f0fa758e902390d471e580770c87dd35de not found: ID does not exist"
Sep 05 20:13:02.796385 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.796355    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:13:02.798946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.798920    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-14a23e-openshift-default-575ddd45dd-g7zph"]
Sep 05 20:13:02.909914 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:02.909891    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:03.042734 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.042645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca\") pod \"785b10c6-8470-4d5a-9cce-ed579a31f06a\" (UID: \"785b10c6-8470-4d5a-9cce-ed579a31f06a\") "
Sep 05 20:13:03.043070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.043045    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "785b10c6-8470-4d5a-9cce-ed579a31f06a" (UID: "785b10c6-8470-4d5a-9cce-ed579a31f06a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:13:03.043834 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.043814    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" path="/var/lib/kubelet/pods/ac8cd9ab-d4e1-4872-b862-e12376529b91/volumes"
Sep 05 20:13:03.143906 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.143697    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/785b10c6-8470-4d5a-9cce-ed579a31f06a-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:03.778613 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.778586    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"
Sep 05 20:13:03.779038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.778626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7" event={"ID":"785b10c6-8470-4d5a-9cce-ed579a31f06a","Type":"ContainerDied","Data":"ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a"}
Sep 05 20:13:03.779038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:03.778685    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff43bfda991394fd0170f78c5cf0b0d321ff239b96798fdc38cb5648d2ccf76a"
Sep 05 20:13:11.089845 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:11.089814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:13:11.096601 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:11.096579    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:13:15.783914 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:15.783875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:15.884042 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:15.884007    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:15.973359 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:15.973327    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"]
Sep 05 20:13:15.976125 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:15.976098    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0f6b5-96xr7"]
Sep 05 20:13:15.976883 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:15.976856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:16.037551 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:16.037464    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:16.345541 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:16.345511    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:16.421423 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:16.421390    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:13:17.043404 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:17.043367    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="785b10c6-8470-4d5a-9cce-ed579a31f06a" path="/var/lib/kubelet/pods/785b10c6-8470-4d5a-9cce-ed579a31f06a/volumes"
Sep 05 20:13:21.347506 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:21.347472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:13:21.418800 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:21.418761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:25.930768 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:25.930734    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:13:36.367908 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.367849    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" podUID="45433a13-16ca-472b-8ff7-a79c2cd4991e" containerName="main" containerID="cri-o://3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9" gracePeriod=45
Sep 05 20:13:36.614835 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.614810    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:13:36.662306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662220    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662278    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662306    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9flmg\" (UniqueName: \"kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662362    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662429    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662499    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home\") pod \"45433a13-16ca-472b-8ff7-a79c2cd4991e\" (UID: \"45433a13-16ca-472b-8ff7-a79c2cd4991e\") "
Sep 05 20:13:36.662754 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662574    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache" (OuterVolumeSpecName: "model-cache") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:36.662813 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662767    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home" (OuterVolumeSpecName: "home") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:36.662976 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662909    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-model-cache\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.662976 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.662935    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-home\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.663180 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.663000    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:36.664964 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.664939    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm" (OuterVolumeSpecName: "dshm") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:13:36.665083 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.664959    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:13:36.665253 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.665231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg" (OuterVolumeSpecName: "kube-api-access-9flmg") pod "45433a13-16ca-472b-8ff7-a79c2cd4991e" (UID: "45433a13-16ca-472b-8ff7-a79c2cd4991e"). InnerVolumeSpecName "kube-api-access-9flmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:13:36.763903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.763871    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45433a13-16ca-472b-8ff7-a79c2cd4991e-tls-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.763903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.763898    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flmg\" (UniqueName: \"kubernetes.io/projected/45433a13-16ca-472b-8ff7-a79c2cd4991e-kube-api-access-9flmg\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.763903 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.763909    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-tmp-dir\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.764142 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.763918    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45433a13-16ca-472b-8ff7-a79c2cd4991e-dshm\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:13:36.950465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.950369    2575 generic.go:356] "Generic (PLEG): container finished" podID="45433a13-16ca-472b-8ff7-a79c2cd4991e" containerID="3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9" exitCode=0
Sep 05 20:13:36.950465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.950411    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"
Sep 05 20:13:36.950668 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.950458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" event={"ID":"45433a13-16ca-472b-8ff7-a79c2cd4991e","Type":"ContainerDied","Data":"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"}
Sep 05 20:13:36.950668 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.950499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9" event={"ID":"45433a13-16ca-472b-8ff7-a79c2cd4991e","Type":"ContainerDied","Data":"46afb88494e37f769ef4e0c43235643fb832e3f16eb594a78f1e5d68b197aa1e"}
Sep 05 20:13:36.950668 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.950517    2575 scope.go:117] "RemoveContainer" containerID="3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"
Sep 05 20:13:36.959803 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.959787    2575 scope.go:117] "RemoveContainer" containerID="3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"
Sep 05 20:13:36.960069 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:13:36.960051    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9\": container with ID starting with 3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9 not found: ID does not exist" containerID="3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"
Sep 05 20:13:36.960110 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.960078    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9"} err="failed to get container status \"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9\": rpc error: code = NotFound desc = could not find container \"3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9\": container with ID starting with 3810bed2795f9a0a0d3dc0ae601e04bddeb0b979a1a8b6e0323b9286f1708cf9 not found: ID does not exist"
Sep 05 20:13:36.971764 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.971738    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:13:36.975775 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:36.975752    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d0f6b5/ipp-route-d0f6b5-kserve-799758fd55-tkkq9"]
Sep 05 20:13:37.043305 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:37.043276    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45433a13-16ca-472b-8ff7-a79c2cd4991e" path="/var/lib/kubelet/pods/45433a13-16ca-472b-8ff7-a79c2cd4991e/volumes"
Sep 05 20:13:56.375500 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:56.375465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:58.226061 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:58.226022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:13:59.050222 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:13:59.050176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:04.620989 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:04.620947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:04.722136 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:04.722095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:04.875380 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:04.875295    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:14:04.875616 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:04.875576    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy" containerID="cri-o://07abd24c074b4133a9e82e85ea3e81574f191bdb392a8b44ee5804002e99af8d" gracePeriod=30
Sep 05 20:14:04.894634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:04.894600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:05.234786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.234698    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"]
Sep 05 20:14:05.235070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235058    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:05.235113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235072    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:05.235113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235102    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45433a13-16ca-472b-8ff7-a79c2cd4991e" containerName="main"
Sep 05 20:14:05.235113 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235107    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45433a13-16ca-472b-8ff7-a79c2cd4991e" containerName="main"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235118    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785b10c6-8470-4d5a-9cce-ed579a31f06a" containerName="revoke-keys"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235124    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785b10c6-8470-4d5a-9cce-ed579a31f06a" containerName="revoke-keys"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235176    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45433a13-16ca-472b-8ff7-a79c2cd4991e" containerName="main"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235197    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235203    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:05.235254 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.235210    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785b10c6-8470-4d5a-9cce-ed579a31f06a" containerName="revoke-keys"
Sep 05 20:14:05.237172 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.237155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:05.240073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.240050    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:14:05.240241 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.240090    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:14:05.244884 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.244851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"]
Sep 05 20:14:05.334380 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.334337    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw\" (UID: \"8a073fa0-463c-4956-b7a1-7c0d1e52890b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:05.435364 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.435326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw\" (UID: \"8a073fa0-463c-4956-b7a1-7c0d1e52890b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:05.435940 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.435920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw\" (UID: \"8a073fa0-463c-4956-b7a1-7c0d1e52890b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:05.548171 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.548084    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:05.589139 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.589106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:05.819216 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:05.819177    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"]
Sep 05 20:14:05.821508 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:14:05.821479    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a073fa0_463c_4956_b7a1_7c0d1e52890b.slice/crio-648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f WatchSource:0}: Error finding container 648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f: Status 404 returned error can't find the container with id 648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f
Sep 05 20:14:06.095833 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:06.095803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw" event={"ID":"8a073fa0-463c-4956-b7a1-7c0d1e52890b","Type":"ContainerStarted","Data":"a3629c73af55ca0b0484625b8f3e6fe464b8c7be1107102ea51f63d7450fdb92"}
Sep 05 20:14:06.095965 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:06.095851    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw" event={"ID":"8a073fa0-463c-4956-b7a1-7c0d1e52890b","Type":"ContainerStarted","Data":"648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f"}
Sep 05 20:14:06.112333 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:06.112278    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw" podStartSLOduration=1.112261865 podStartE2EDuration="1.112261865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:14:06.111226072 +0000 UTC m=+1855.528299499" watchObservedRunningTime="2026-09-05 20:14:06.112261865 +0000 UTC m=+1855.529335323"
Sep 05 20:14:07.101943 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:07.101909    2575 generic.go:356] "Generic (PLEG): container finished" podID="8a073fa0-463c-4956-b7a1-7c0d1e52890b" containerID="a3629c73af55ca0b0484625b8f3e6fe464b8c7be1107102ea51f63d7450fdb92" exitCode=0
Sep 05 20:14:07.102395 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:07.101991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw" event={"ID":"8a073fa0-463c-4956-b7a1-7c0d1e52890b","Type":"ContainerDied","Data":"a3629c73af55ca0b0484625b8f3e6fe464b8c7be1107102ea51f63d7450fdb92"}
Sep 05 20:14:08.236374 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:08.236346    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:08.258946 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:08.258915    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca\") pod \"8a073fa0-463c-4956-b7a1-7c0d1e52890b\" (UID: \"8a073fa0-463c-4956-b7a1-7c0d1e52890b\") "
Sep 05 20:14:08.259293 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:08.259267    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a073fa0-463c-4956-b7a1-7c0d1e52890b" (UID: "8a073fa0-463c-4956-b7a1-7c0d1e52890b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:08.360243 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:08.360129    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073fa0-463c-4956-b7a1-7c0d1e52890b-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:09.113830 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:09.113801    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"
Sep 05 20:14:09.113830 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:09.113815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw" event={"ID":"8a073fa0-463c-4956-b7a1-7c0d1e52890b","Type":"ContainerDied","Data":"648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f"}
Sep 05 20:14:09.114038 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:09.113857    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="648e3eded81023a3d224ffa3a77d6e720ef294f3064c5c84f4070b8f3cc40a1f"
Sep 05 20:14:10.120873 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.120839    2575 generic.go:356] "Generic (PLEG): container finished" podID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerID="07abd24c074b4133a9e82e85ea3e81574f191bdb392a8b44ee5804002e99af8d" exitCode=0
Sep 05 20:14:10.121248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.120910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerDied","Data":"07abd24c074b4133a9e82e85ea3e81574f191bdb392a8b44ee5804002e99af8d"}
Sep 05 20:14:10.121248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.120961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl" event={"ID":"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8","Type":"ContainerDied","Data":"81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca"}
Sep 05 20:14:10.121248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.120978    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ff3ec76170f8a9a95e58cdf524df3e81c67e77ba5ce45484de052b80aa6eca"
Sep 05 20:14:10.121248 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.120995    2575 scope.go:117] "RemoveContainer" containerID="07910360db609348b6db00860fcd317fae58fff09560f409a5fb24a2346a55bb"
Sep 05 20:14:10.123163 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.123144    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:14:10.177970 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.177893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.177970 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.177926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8cwt\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.177982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.177996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178018    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178051    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178088    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178225 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:10.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178269    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178313    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:10.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:10.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178373    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy\") pod \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\" (UID: \"0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8\") "
Sep 05 20:14:10.178545 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178622    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178743    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178760    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178774    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178787    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.178809 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.178801    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.180450 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.180423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt" (OuterVolumeSpecName: "kube-api-access-j8cwt") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "kube-api-access-j8cwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:10.180568 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.180550    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:10.180722 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.180697    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:14:10.180813 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.180703    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" (UID: "0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:10.280051 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.280023    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.280051 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.280048    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.280051 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.280057    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:10.280310 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:10.280067    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8cwt\" (UniqueName: \"kubernetes.io/projected/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8-kube-api-access-j8cwt\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:11.127264 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:11.127233    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"
Sep 05 20:14:11.162354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:11.162321    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:14:11.166385 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:11.166355    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0f6b5-openshift-default-6887c4c5dd-4tgpl"]
Sep 05 20:14:13.043027 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:13.042993    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" path="/var/lib/kubelet/pods/0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8/volumes"
Sep 05 20:14:15.684662 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.684625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:15.786131 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.786071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:15.882977 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.882944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:15.930984 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.930955    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"]
Sep 05 20:14:15.935520 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.935457    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6693ae-fqjsw"]
Sep 05 20:14:15.953319 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:15.953287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:16.317075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:16.317037    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:17.043808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:17.043771    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a073fa0-463c-4956-b7a1-7c0d1e52890b" path="/var/lib/kubelet/pods/8a073fa0-463c-4956-b7a1-7c0d1e52890b/volumes"
Sep 05 20:14:19.426859 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:19.426820    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:14:22.291373 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.291331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:22.391818 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.391786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:22.545710 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.545632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:14:22.545948 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.545923    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy" containerID="cri-o://a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02" gracePeriod=30
Sep 05 20:14:22.563319 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.563293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:22.892462 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892422    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"]
Sep 05 20:14:22.892804 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892791    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:22.892853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892805    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:22.892853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892816    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:22.892853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892822    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8cd9ab-d4e1-4872-b862-e12376529b91" containerName="istio-proxy"
Sep 05 20:14:22.892853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892832    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a073fa0-463c-4956-b7a1-7c0d1e52890b" containerName="revoke-keys"
Sep 05 20:14:22.892853 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892837    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a073fa0-463c-4956-b7a1-7c0d1e52890b" containerName="revoke-keys"
Sep 05 20:14:22.893002 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892899    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:22.893002 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.892908    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a073fa0-463c-4956-b7a1-7c0d1e52890b" containerName="revoke-keys"
Sep 05 20:14:22.894998 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.894982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:22.897702 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.897682    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:14:22.897824 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.897749    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:14:22.903634 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.903609    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"]
Sep 05 20:14:22.996141 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.996103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r\" (UID: \"7eece191-d31c-4e75-a499-370734b6b6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:22.996653 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:22.996633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:23.097612 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:23.097573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r\" (UID: \"7eece191-d31c-4e75-a499-370734b6b6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:23.098181 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:23.098160    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r\" (UID: \"7eece191-d31c-4e75-a499-370734b6b6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:23.206445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:23.206362    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:23.497923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:23.497895    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"]
Sep 05 20:14:23.500443 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:14:23.500411    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eece191_d31c_4e75_a499_370734b6b6d8.slice/crio-ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc WatchSource:0}: Error finding container ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc: Status 404 returned error can't find the container with id ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc
Sep 05 20:14:24.191708 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:24.191679    2575 generic.go:356] "Generic (PLEG): container finished" podID="7eece191-d31c-4e75-a499-370734b6b6d8" containerID="640e77c098f93fb639ed3e785ec4e67102f19698016492747c171e37fb6d8be9" exitCode=0
Sep 05 20:14:24.191847 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:24.191759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r" event={"ID":"7eece191-d31c-4e75-a499-370734b6b6d8","Type":"ContainerDied","Data":"640e77c098f93fb639ed3e785ec4e67102f19698016492747c171e37fb6d8be9"}
Sep 05 20:14:24.191847 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:24.191811    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r" event={"ID":"7eece191-d31c-4e75-a499-370734b6b6d8","Type":"ContainerStarted","Data":"ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc"}
Sep 05 20:14:24.301707 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:24.301670    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:25.335797 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.335767    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:25.426504 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.426465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca\") pod \"7eece191-d31c-4e75-a499-370734b6b6d8\" (UID: \"7eece191-d31c-4e75-a499-370734b6b6d8\") "
Sep 05 20:14:25.426825 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.426803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7eece191-d31c-4e75-a499-370734b6b6d8" (UID: "7eece191-d31c-4e75-a499-370734b6b6d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:25.527430 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.527349    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eece191-d31c-4e75-a499-370734b6b6d8-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:25.691104 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.691070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:25.792341 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:25.792244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:26.202475 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:26.202443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r" event={"ID":"7eece191-d31c-4e75-a499-370734b6b6d8","Type":"ContainerDied","Data":"ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc"}
Sep 05 20:14:26.202630 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:26.202479    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"
Sep 05 20:14:26.202630 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:26.202488    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca57e249fa34c972c482304b1ba315ae8f602b4028a01e0787deb27e2df42ccc"
Sep 05 20:14:27.788359 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.788335    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:14:27.951692 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951655    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951692 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951693    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951713    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951754    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951807    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951896    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.951955 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.952304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.951968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket\") pod \"729835be-311b-4a09-a4c9-360c23e18856\" (UID: \"729835be-311b-4a09-a4c9-360c23e18856\") "
Sep 05 20:14:27.952304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:27.952304 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952085    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952287    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data" (OuterVolumeSpecName: "istio-data") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952293    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952418    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952435    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952449    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:27.952491 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.952461    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:27.954087 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.954061    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:14:27.954212 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.954115    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs" (OuterVolumeSpecName: "kube-api-access-ktkfs") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "kube-api-access-ktkfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:27.954212 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.954179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token" (OuterVolumeSpecName: "istio-token") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:27.954328 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:27.954247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "729835be-311b-4a09-a4c9-360c23e18856" (UID: "729835be-311b-4a09-a4c9-360c23e18856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:28.053198 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.053162    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:28.053362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.053217    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/729835be-311b-4a09-a4c9-360c23e18856-kube-api-access-ktkfs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:28.053362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.053228    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729835be-311b-4a09-a4c9-360c23e18856-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:28.053362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.053236    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729835be-311b-4a09-a4c9-360c23e18856-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:28.053362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.053244    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729835be-311b-4a09-a4c9-360c23e18856-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:28.213744 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.213655    2575 generic.go:356] "Generic (PLEG): container finished" podID="729835be-311b-4a09-a4c9-360c23e18856" containerID="a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02" exitCode=0
Sep 05 20:14:28.213744 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.213699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerDied","Data":"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"}
Sep 05 20:14:28.213744 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.213707    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"
Sep 05 20:14:28.213744 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.213750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b" event={"ID":"729835be-311b-4a09-a4c9-360c23e18856","Type":"ContainerDied","Data":"02833d31df9ad5d64c14e3a4c886b83c3b861984564369ae0910c73eb238cb34"}
Sep 05 20:14:28.214050 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.213773    2575 scope.go:117] "RemoveContainer" containerID="a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"
Sep 05 20:14:28.223447 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.223429    2575 scope.go:117] "RemoveContainer" containerID="5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"
Sep 05 20:14:28.231140 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.231124    2575 scope.go:117] "RemoveContainer" containerID="a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"
Sep 05 20:14:28.231891 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:14:28.231868    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02\": container with ID starting with a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02 not found: ID does not exist" containerID="a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"
Sep 05 20:14:28.231993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.231896    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02"} err="failed to get container status \"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02\": rpc error: code = NotFound desc = could not find container \"a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02\": container with ID starting with a08d8bfc57421000be5e4307723e7a8d02bb7faac5fc3462c39350dcf44eea02 not found: ID does not exist"
Sep 05 20:14:28.231993 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.231913    2575 scope.go:117] "RemoveContainer" containerID="5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"
Sep 05 20:14:28.232218 ip-10-0-129-128 kubenswrapper[2575]: E0905 20:14:28.232197    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526\": container with ID starting with 5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526 not found: ID does not exist" containerID="5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"
Sep 05 20:14:28.232296 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.232222    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526"} err="failed to get container status \"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526\": rpc error: code = NotFound desc = could not find container \"5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526\": container with ID starting with 5d0bd71a3ec7d0c62d8fcc50c2e2f4c1c1537c36049e5db0d89662268d84b526 not found: ID does not exist"
Sep 05 20:14:28.235767 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.235743    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:14:28.240255 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:28.240232    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6693ae-openshift-default-74b5d4c7b-p229b"]
Sep 05 20:14:29.042966 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:29.042932    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729835be-311b-4a09-a4c9-360c23e18856" path="/var/lib/kubelet/pods/729835be-311b-4a09-a4c9-360c23e18856/volumes"
Sep 05 20:14:33.181512 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:33.181479    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:33.837602 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:33.837568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:33.923808 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:33.923772    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:33.996736 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:33.996700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:34.047923 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:34.047891    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"]
Sep 05 20:14:34.051027 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:34.051002    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4d1885-h4g8r"]
Sep 05 20:14:34.064730 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:34.064699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:34.420512 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:34.420477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:35.043082 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:35.043049    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eece191-d31c-4e75-a499-370734b6b6d8" path="/var/lib/kubelet/pods/7eece191-d31c-4e75-a499-370734b6b6d8/volumes"
Sep 05 20:14:36.727505 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:36.727470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:14:40.623053 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:40.623012    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:40.723379 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:40.723343    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:40.899492 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:40.899399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:41.257708 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.257614    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"]
Sep 05 20:14:41.258028 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258013    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:14:41.258073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258029    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:14:41.258073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258047    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:14:41.258073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258054    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:14:41.258073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258066    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eece191-d31c-4e75-a499-370734b6b6d8" containerName="revoke-keys"
Sep 05 20:14:41.258073 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258072    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eece191-d31c-4e75-a499-370734b6b6d8" containerName="revoke-keys"
Sep 05 20:14:41.258262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258080    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:41.258262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258086    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:41.258262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258143    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:14:41.258262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258152    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0089cdc3-4e30-46d3-ab7e-e5dcbac2cec8" containerName="istio-proxy"
Sep 05 20:14:41.258262 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.258159    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eece191-d31c-4e75-a499-370734b6b6d8" containerName="revoke-keys"
Sep 05 20:14:41.262905 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.262889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:41.265729 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.265704    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:14:41.266996 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.266928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:14:41.269168 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.269144    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"]
Sep 05 20:14:41.374603 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.374572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc\" (UID: \"ebe565ac-4779-4f24-8e4c-80aea0adbb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:41.378690 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.378666    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:41.475194 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.475162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc\" (UID: \"ebe565ac-4779-4f24-8e4c-80aea0adbb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:41.475879 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.475844    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc\" (UID: \"ebe565ac-4779-4f24-8e4c-80aea0adbb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:41.574281 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.574241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:41.849624 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:41.849599    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"]
Sep 05 20:14:41.851943 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:14:41.851914    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe565ac_4779_4f24_8e4c_80aea0adbb45.slice/crio-25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3 WatchSource:0}: Error finding container 25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3: Status 404 returned error can't find the container with id 25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3
Sep 05 20:14:42.290116 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:42.290079    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc" event={"ID":"ebe565ac-4779-4f24-8e4c-80aea0adbb45","Type":"ContainerStarted","Data":"8a82cec7bffd944c7bba357636daba70c6019f06569b0c50e9240f11b045899a"}
Sep 05 20:14:42.290319 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:42.290145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc" event={"ID":"ebe565ac-4779-4f24-8e4c-80aea0adbb45","Type":"ContainerStarted","Data":"25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3"}
Sep 05 20:14:42.308502 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:42.308453    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc" podStartSLOduration=1.308436945 podStartE2EDuration="1.308436945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:14:42.306273025 +0000 UTC m=+1891.723346451" watchObservedRunningTime="2026-09-05 20:14:42.308436945 +0000 UTC m=+1891.725510369"
Sep 05 20:14:43.303401 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:43.303364    2575 generic.go:356] "Generic (PLEG): container finished" podID="ebe565ac-4779-4f24-8e4c-80aea0adbb45" containerID="8a82cec7bffd944c7bba357636daba70c6019f06569b0c50e9240f11b045899a" exitCode=0
Sep 05 20:14:43.303791 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:43.303440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc" event={"ID":"ebe565ac-4779-4f24-8e4c-80aea0adbb45","Type":"ContainerDied","Data":"8a82cec7bffd944c7bba357636daba70c6019f06569b0c50e9240f11b045899a"}
Sep 05 20:14:44.435842 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:44.435818    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:44.508635 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:44.508595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca\") pod \"ebe565ac-4779-4f24-8e4c-80aea0adbb45\" (UID: \"ebe565ac-4779-4f24-8e4c-80aea0adbb45\") "
Sep 05 20:14:44.508973 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:44.508944    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebe565ac-4779-4f24-8e4c-80aea0adbb45" (UID: "ebe565ac-4779-4f24-8e4c-80aea0adbb45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:44.609715 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:44.609627    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe565ac-4779-4f24-8e4c-80aea0adbb45-maas-api-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:14:44.631399 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:44.631364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:45.314862 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:45.314830    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc" event={"ID":"ebe565ac-4779-4f24-8e4c-80aea0adbb45","Type":"ContainerDied","Data":"25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3"}
Sep 05 20:14:45.314862 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:45.314860    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"
Sep 05 20:14:45.315074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:45.314876    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25689d4225409ea65369f788a2bf1629ff7b3e237cb2b41dfdef3b85158faab3"
Sep 05 20:14:51.738708 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.738670    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:51.811277 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.811237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:51.874662 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.874625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:51.916544 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.916510    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"]
Sep 05 20:14:51.921313 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.921286    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0cc31a-psxkc"]
Sep 05 20:14:51.936722 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:51.936689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:52.283032 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:52.282991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:53.042161 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:53.042128    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe565ac-4779-4f24-8e4c-80aea0adbb45" path="/var/lib/kubelet/pods/ebe565ac-4779-4f24-8e4c-80aea0adbb45/volumes"
Sep 05 20:14:54.244533 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:54.244498    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:14:59.042577 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:59.042543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:59.146971 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:59.146934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:14:59.283516 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:59.283484    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:14:59.283760 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:59.283738    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy" containerID="cri-o://2b5e19c05528619b71a2dde3fc44da9cf5ab72b3f0499219dd79aca865cd873a" gracePeriod=30
Sep 05 20:14:59.306305 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:14:59.306181    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:15:03.430911 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:03.430879    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:15:04.413354 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.413322    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerID="2b5e19c05528619b71a2dde3fc44da9cf5ab72b3f0499219dd79aca865cd873a" exitCode=0
Sep 05 20:15:04.413477 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.413391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerDied","Data":"2b5e19c05528619b71a2dde3fc44da9cf5ab72b3f0499219dd79aca865cd873a"}
Sep 05 20:15:04.413477 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.413454    2575 scope.go:117] "RemoveContainer" containerID="5c8acf6b3ae4776197c1aa38c6434775f0ec28ef390485bf2e4a8b7a2a898328"
Sep 05 20:15:04.547331 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.547307    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:15:04.695415 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695376    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695433    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695460    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695485    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695503    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6rgh\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695614 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695630    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695663    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert\") pod \"d9febecf-a2dc-41cf-b397-8dadc7519b55\" (UID: \"d9febecf-a2dc-41cf-b397-8dadc7519b55\") "
Sep 05 20:15:04.695941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695674    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:15:04.695941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695855    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:15:04.695941 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695912    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:15:04.696218 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695958    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.696218 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.695977    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-workload-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.696218 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.696118    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:15:04.696336 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.696242    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:15:04.697819 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.697798    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:15:04.697964 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.697913    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:15:04.698197 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.698159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:15:04.698259 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.698218    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh" (OuterVolumeSpecName: "kube-api-access-n6rgh") pod "d9febecf-a2dc-41cf-b397-8dadc7519b55" (UID: "d9febecf-a2dc-41cf-b397-8dadc7519b55"). InnerVolumeSpecName "kube-api-access-n6rgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:15:04.796537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796500    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-token\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796528    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-data\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796537    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-credential-socket\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796537 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796547    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6rgh\" (UniqueName: \"kubernetes.io/projected/d9febecf-a2dc-41cf-b397-8dadc7519b55-kube-api-access-n6rgh\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796555    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-podinfo\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796563    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9febecf-a2dc-41cf-b397-8dadc7519b55-istio-envoy\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:04.796786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:04.796571    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9febecf-a2dc-41cf-b397-8dadc7519b55-istiod-ca-cert\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:15:05.419939 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:05.419903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv" event={"ID":"d9febecf-a2dc-41cf-b397-8dadc7519b55","Type":"ContainerDied","Data":"5c414fc10880d0d3982cc743cac128317a243718e8b6c74b8a7cf1c441516e7c"}
Sep 05 20:15:05.420117 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:05.419947    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"
Sep 05 20:15:05.420117 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:05.419988    2575 scope.go:117] "RemoveContainer" containerID="2b5e19c05528619b71a2dde3fc44da9cf5ab72b3f0499219dd79aca865cd873a"
Sep 05 20:15:05.439609 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:05.439585    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:15:05.443556 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:05.443532    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0cc31a-openshift-default-5fb54df9d4-8v9rv"]
Sep 05 20:15:07.043203 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:15:07.043157    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" path="/var/lib/kubelet/pods/d9febecf-a2dc-41cf-b397-8dadc7519b55/volumes"
Sep 05 20:16:22.840149 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:22.840113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:16:23.626158 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:23.626122    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:16:24.710815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:24.710780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:16:26.277035 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:26.277000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:16:50.335211 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:50.335152    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:16:58.019535 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:16:58.019495    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:17:02.172306 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:02.172260    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:17:02.227239 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:02.227204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:17:05.327550 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:05.327511    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:17:11.363621 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:11.363582    2575 scope.go:117] "RemoveContainer" containerID="07abd24c074b4133a9e82e85ea3e81574f191bdb392a8b44ee5804002e99af8d"
Sep 05 20:17:12.023767 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:12.023733    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:17:20.625316 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:20.625279    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:17:20.728734 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:20.728694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:17:24.225093 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:24.225059    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:17:25.146328 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:25.146274    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:17:34.448503 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:17:34.448464    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:18:11.118789 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:18:11.118751    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:18:11.128735 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:18:11.128711    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:19:02.123936 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:02.123896    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:19:09.021352 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:09.021272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:19:11.389769 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:11.389736    2575 scope.go:117] "RemoveContainer" containerID="766507f483f953791e727e66b4c026fb432c0b3f2cf34b53e7bab8cf598fc31a"
Sep 05 20:19:16.315149 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:16.315115    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:19:23.277649 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:23.277609    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:19:25.720325 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:25.720287    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:19:36.839362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:36.839320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:19:40.021325 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:40.021288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:19:59.939459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:19:59.939418    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:20:01.139068 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:01.139031    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:20:02.917868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:02.917833    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:11.215074 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:11.215037    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:11.410683 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:11.410652    2575 scope.go:117] "RemoveContainer" containerID="a3629c73af55ca0b0484625b8f3e6fe464b8c7be1107102ea51f63d7450fdb92"
Sep 05 20:20:17.923302 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:17.923264    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:18.358047 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358014    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:18.358398 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358385    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe565ac-4779-4f24-8e4c-80aea0adbb45" containerName="revoke-keys"
Sep 05 20:20:18.358458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358400    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe565ac-4779-4f24-8e4c-80aea0adbb45" containerName="revoke-keys"
Sep 05 20:20:18.358458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:18.358458 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358420    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:18.358562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358478    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:18.358562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358485    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe565ac-4779-4f24-8e4c-80aea0adbb45" containerName="revoke-keys"
Sep 05 20:20:18.358562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358492    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="729835be-311b-4a09-a4c9-360c23e18856" containerName="istio-proxy"
Sep 05 20:20:18.358562 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.358499    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:18.361413 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.361398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.365004 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.364981    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 05 20:20:18.365122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.365011    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 05 20:20:18.365122 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.365013    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:20:18.365257 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.365125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p9ckr\""
Sep 05 20:20:18.365257 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.365125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 05 20:20:18.370724 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.370692    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:18.527617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.527584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.527617 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.527621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lss\" (UniqueName: \"kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.527861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.527649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.527861 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.527781    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.628421 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.628347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.628542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.628418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.628542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.628454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.628542 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.628482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lss\" (UniqueName: \"kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.628983 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.628962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.630793 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.630771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.630890 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.630867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.635361 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.635342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lss\" (UniqueName: \"kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss\") pod \"maas-api-7b97c6dbfb-bt2kx\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") " pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.672524 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.672496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:18.942868 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.942791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:18.946068 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:20:18.946035    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd32cc2_c3db_4c62_88bb_0c4c38ee9ec7.slice/crio-16ff00a9ba95147882dce4b446723960a4789f07c456e3a546ff4c316745ac04 WatchSource:0}: Error finding container 16ff00a9ba95147882dce4b446723960a4789f07c456e3a546ff4c316745ac04: Status 404 returned error can't find the container with id 16ff00a9ba95147882dce4b446723960a4789f07c456e3a546ff4c316745ac04
Sep 05 20:20:18.947454 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:18.947433    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 20:20:19.936202 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:19.936155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" event={"ID":"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7","Type":"ContainerStarted","Data":"ff370601f91e9f4aa19a7b06ba714d745d698f1c99825f766e2cf38b6ba19cec"}
Sep 05 20:20:19.936386 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:19.936240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:19.936386 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:19.936263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" event={"ID":"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7","Type":"ContainerStarted","Data":"16ff00a9ba95147882dce4b446723960a4789f07c456e3a546ff4c316745ac04"}
Sep 05 20:20:19.952877 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:19.952821    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" podStartSLOduration=1.471018558 podStartE2EDuration="1.952805182s" totalImagesPullingTime="481.786624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:20:19.951515195 +0000 UTC m=+2229.368588611" watchObservedRunningTime="2026-09-05 20:20:19.952805182 +0000 UTC m=+2229.369878607"
Sep 05 20:20:25.744741 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.744700    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:25.745216 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.744933    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" podUID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" containerName="maas-api" containerID="cri-o://ff370601f91e9f4aa19a7b06ba714d745d698f1c99825f766e2cf38b6ba19cec" gracePeriod=30
Sep 05 20:20:25.750166 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.750143    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:25.773045 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.773018    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-htt5w"]
Sep 05 20:20:25.773419 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.773405    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:25.773476 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.773420    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9febecf-a2dc-41cf-b397-8dadc7519b55" containerName="istio-proxy"
Sep 05 20:20:25.776278 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.776261    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.784786 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.784760    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-htt5w"]
Sep 05 20:20:25.790034 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.790010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwct4\" (UniqueName: \"kubernetes.io/projected/bb668aee-6435-4eb9-b868-23b600dc0d96-kube-api-access-gwct4\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.790157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.790043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-maas-api-tls\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.790157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.790065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb668aee-6435-4eb9-b868-23b600dc0d96-openshift-service-ca\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.790157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.790084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-metrics-certs\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.817878 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.817839    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:25.890819 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.890778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwct4\" (UniqueName: \"kubernetes.io/projected/bb668aee-6435-4eb9-b868-23b600dc0d96-kube-api-access-gwct4\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.890994 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.890909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-maas-api-tls\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.890994 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.890945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb668aee-6435-4eb9-b868-23b600dc0d96-openshift-service-ca\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.890994 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.890974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-metrics-certs\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.891815 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.891788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb668aee-6435-4eb9-b868-23b600dc0d96-openshift-service-ca\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.893867 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.893841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-maas-api-tls\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.894035 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.894014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb668aee-6435-4eb9-b868-23b600dc0d96-metrics-certs\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.900493 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.900468    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwct4\" (UniqueName: \"kubernetes.io/projected/bb668aee-6435-4eb9-b868-23b600dc0d96-kube-api-access-gwct4\") pod \"maas-api-6886844f88-htt5w\" (UID: \"bb668aee-6435-4eb9-b868-23b600dc0d96\") " pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:25.966332 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.966299    2575 generic.go:356] "Generic (PLEG): container finished" podID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" containerID="ff370601f91e9f4aa19a7b06ba714d745d698f1c99825f766e2cf38b6ba19cec" exitCode=0
Sep 05 20:20:25.966487 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.966350    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" event={"ID":"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7","Type":"ContainerDied","Data":"ff370601f91e9f4aa19a7b06ba714d745d698f1c99825f766e2cf38b6ba19cec"}
Sep 05 20:20:25.990201 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:25.990159    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:26.092885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.092847    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca\") pod \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") "
Sep 05 20:20:26.092885 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.092891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55lss\" (UniqueName: \"kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss\") pod \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") "
Sep 05 20:20:26.093151 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.092934    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls\") pod \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") "
Sep 05 20:20:26.093151 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.092958    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs\") pod \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\" (UID: \"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7\") "
Sep 05 20:20:26.093348 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.093308    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" (UID: "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:20:26.095075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.095050    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" (UID: "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:20:26.095075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.095062    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" (UID: "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:20:26.095227 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.095094    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss" (OuterVolumeSpecName: "kube-api-access-55lss") pod "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" (UID: "dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7"). InnerVolumeSpecName "kube-api-access-55lss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:20:26.123346 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.123311    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:26.194500 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.194440    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-openshift-service-ca\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:20:26.194835 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.194475    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lss\" (UniqueName: \"kubernetes.io/projected/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-kube-api-access-55lss\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:20:26.194835 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.194793    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-maas-api-tls\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:20:26.194835 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.194812    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7-metrics-certs\") on node \"ip-10-0-129-128.ec2.internal\" DevicePath \"\""
Sep 05 20:20:26.393484 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.393459    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886844f88-htt5w"]
Sep 05 20:20:26.395743 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:20:26.395713    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb668aee_6435_4eb9_b868_23b600dc0d96.slice/crio-ca505294d14b6e22d753c7375e6b670de7807692898f90891ddd62a8b916a28d WatchSource:0}: Error finding container ca505294d14b6e22d753c7375e6b670de7807692898f90891ddd62a8b916a28d: Status 404 returned error can't find the container with id ca505294d14b6e22d753c7375e6b670de7807692898f90891ddd62a8b916a28d
Sep 05 20:20:26.972543 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.972507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w" event={"ID":"bb668aee-6435-4eb9-b868-23b600dc0d96","Type":"ContainerStarted","Data":"9c370ff9fda4190f16b67458ab99a4a8a3c18afcc268d06bf17922de8a77024e"}
Sep 05 20:20:26.972987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.972590    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w" event={"ID":"bb668aee-6435-4eb9-b868-23b600dc0d96","Type":"ContainerStarted","Data":"ca505294d14b6e22d753c7375e6b670de7807692898f90891ddd62a8b916a28d"}
Sep 05 20:20:26.972987 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.972627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:26.973986 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.973963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx" event={"ID":"dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7","Type":"ContainerDied","Data":"16ff00a9ba95147882dce4b446723960a4789f07c456e3a546ff4c316745ac04"}
Sep 05 20:20:26.974036 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.974015    2575 scope.go:117] "RemoveContainer" containerID="ff370601f91e9f4aa19a7b06ba714d745d698f1c99825f766e2cf38b6ba19cec"
Sep 05 20:20:26.974075 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.973972    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"
Sep 05 20:20:26.992806 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:26.992764    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w" podStartSLOduration=1.5596073110000002 podStartE2EDuration="1.992751207s" totalImagesPullingTime="433.143896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:20:26.989757993 +0000 UTC m=+2236.406831417" watchObservedRunningTime="2026-09-05 20:20:26.992751207 +0000 UTC m=+2236.409824630"
Sep 05 20:20:27.003970 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:27.003946    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:27.007279 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:27.007259    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97c6dbfb-bt2kx"]
Sep 05 20:20:27.042397 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:27.042341    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" path="/var/lib/kubelet/pods/dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7/volumes"
Sep 05 20:20:31.918344 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:31.918310    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:32.985394 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:32.985360    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6886844f88-htt5w"
Sep 05 20:20:37.627938 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:37.627901    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:44.720320 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:44.720283    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tgslb"]
Sep 05 20:20:53.120263 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:53.120230    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cjj8p_0cb46996-1381-4b32-9c28-49a2bf161710/server/0.log"
Sep 05 20:20:53.235982 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:53.235947    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xlm5x_1542feed-728f-463c-a706-3e4b5e82c240/manager/0.log"
Sep 05 20:20:53.578342 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:53.578299    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-qbtrn_4c7b57ab-b186-4fea-9a80-fb20df883193/copy-manifests/0.log"
Sep 05 20:20:53.595756 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:53.595717    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-qbtrn_4c7b57ab-b186-4fea-9a80-fb20df883193/manager/0.log"
Sep 05 20:20:53.920382 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:53.920295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6886844f88-htt5w_bb668aee-6435-4eb9-b868-23b600dc0d96/maas-api/0.log"
Sep 05 20:20:55.653410 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:55.653369    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tgslb_3c8ee640-d75c-44cb-9948-06b0c62ece6b/limitador/0.log"
Sep 05 20:20:56.211239 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:56.211208    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ss2lh_21de3aa5-6e75-48cb-9e65-358b8245f35f/discovery/0.log"
Sep 05 20:20:57.390756 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:57.390722    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mvfx7_87e3c99b-b33e-4957-b278-dea511e3acdf/main/0.log"
Sep 05 20:20:57.720043 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:20:57.719961    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-phr4h_4005545a-cb28-4e95-bdf3-6e2fd375d457/main/0.log"
Sep 05 20:21:03.194750 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:03.194712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:21:03.292270 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:03.292232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:21:06.301530 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:06.301486    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xt85s_3a134963-6f04-4a07-ad64-8482ce725d59/global-pull-secret-syncer/0.log"
Sep 05 20:21:06.351150 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:06.351118    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log"
Sep 05 20:21:06.445070 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:06.445038    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-128.ec2.internal_75b7f8ba7b88ef6444d6b7d4c2ce3a73/haproxy/0.log"
Sep 05 20:21:10.959779 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:10.959747    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tgslb_3c8ee640-d75c-44cb-9948-06b0c62ece6b/limitador/0.log"
Sep 05 20:21:11.432264 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:11.432230    2575 scope.go:117] "RemoveContainer" containerID="8a82cec7bffd944c7bba357636daba70c6019f06569b0c50e9240f11b045899a"
Sep 05 20:21:11.440144 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:11.440122    2575 scope.go:117] "RemoveContainer" containerID="640e77c098f93fb639ed3e785ec4e67102f19698016492747c171e37fb6d8be9"
Sep 05 20:21:12.665931 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.665878    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/alertmanager/0.log"
Sep 05 20:21:12.687144 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.687107    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/config-reloader/0.log"
Sep 05 20:21:12.708657 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.708630    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/kube-rbac-proxy-web/0.log"
Sep 05 20:21:12.729445 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.729421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/kube-rbac-proxy/0.log"
Sep 05 20:21:12.752402 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.752381    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/kube-rbac-proxy-metric/0.log"
Sep 05 20:21:12.772378 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.772342    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/prom-label-proxy/0.log"
Sep 05 20:21:12.799127 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:12.799087    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68ed562f-5637-47bc-9156-612e4056d78c/init-config-reloader/0.log"
Sep 05 20:21:13.157807 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.157777    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v74ds_beaf5a30-7efb-4893-8164-d650ce0d64e0/node-exporter/0.log"
Sep 05 20:21:13.178081 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.178051    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v74ds_beaf5a30-7efb-4893-8164-d650ce0d64e0/kube-rbac-proxy/0.log"
Sep 05 20:21:13.200417 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.200389    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v74ds_beaf5a30-7efb-4893-8164-d650ce0d64e0/init-textfile/0.log"
Sep 05 20:21:13.574146 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.574116    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b9b8f9f-j5bff_907f9b51-1f22-42f9-9c35-2bb9142c0180/telemeter-client/0.log"
Sep 05 20:21:13.612650 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.612617    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b9b8f9f-j5bff_907f9b51-1f22-42f9-9c35-2bb9142c0180/reload/0.log"
Sep 05 20:21:13.637430 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.637402    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7b9b8f9f-j5bff_907f9b51-1f22-42f9-9c35-2bb9142c0180/kube-rbac-proxy/0.log"
Sep 05 20:21:13.676897 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.676863    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/thanos-query/0.log"
Sep 05 20:21:13.715042 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.715014    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/kube-rbac-proxy-web/0.log"
Sep 05 20:21:13.737282 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.737259    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/kube-rbac-proxy/0.log"
Sep 05 20:21:13.760774 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.760746    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/prom-label-proxy/0.log"
Sep 05 20:21:13.783554 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.783525    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/kube-rbac-proxy-rules/0.log"
Sep 05 20:21:13.804138 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:13.804103    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f97597988-wtqv4_6ea72caf-299b-449b-b87f-fecf7c9c1084/kube-rbac-proxy-metrics/0.log"
Sep 05 20:21:16.000918 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:16.000883    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/1.log"
Sep 05 20:21:16.010459 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:16.010433    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hl7bh_d07b9fe0-fd1d-447c-9a59-601164d0fcbc/console-operator/2.log"
Sep 05 20:21:16.548826 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:16.548796    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54f6498b86-g2tkf_a74f0648-b125-4547-a0f7-7c8dae9492ca/console/0.log"
Sep 05 20:21:17.195989 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.195954    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"]
Sep 05 20:21:17.196422 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.196318    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" containerName="maas-api"
Sep 05 20:21:17.196422 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.196329    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" containerName="maas-api"
Sep 05 20:21:17.196422 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.196404    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd32cc2-c3db-4c62-88bb-0c4c38ee9ec7" containerName="maas-api"
Sep 05 20:21:17.199362 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.199341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.201650 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.201629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hdj6r\"/\"default-dockercfg-bdppc\""
Sep 05 20:21:17.201865 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.201849    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"openshift-service-ca.crt\""
Sep 05 20:21:17.202513 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.202496    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"kube-root-ca.crt\""
Sep 05 20:21:17.206530 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.206507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"]
Sep 05 20:21:17.251540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.251501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-lib-modules\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.251540 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.251539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-proc\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.251797 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.251570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-sys\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.251797 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.251703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5rn\" (UniqueName: \"kubernetes.io/projected/46fa9287-f9fc-4f17-b7c4-84343a3d764f-kube-api-access-8j5rn\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.251797 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.251758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-podres\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352733 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-podres\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-lib-modules\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-proc\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-sys\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5rn\" (UniqueName: \"kubernetes.io/projected/46fa9287-f9fc-4f17-b7c4-84343a3d764f-kube-api-access-8j5rn\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-podres\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-proc\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-lib-modules\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.352929 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.352927    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fa9287-f9fc-4f17-b7c4-84343a3d764f-sys\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.360517 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.360495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5rn\" (UniqueName: \"kubernetes.io/projected/46fa9287-f9fc-4f17-b7c4-84343a3d764f-kube-api-access-8j5rn\") pod \"perf-node-gather-daemonset-rh2v8\" (UID: \"46fa9287-f9fc-4f17-b7c4-84343a3d764f\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.509952 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.509854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:17.780465 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:17.780436    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"]
Sep 05 20:21:17.782367 ip-10-0-129-128 kubenswrapper[2575]: W0905 20:21:17.782342    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46fa9287_f9fc_4f17_b7c4_84343a3d764f.slice/crio-57d2f98e0f9e8e63c3e581402eec3c188883414d2114df2402087ecc3ab1a633 WatchSource:0}: Error finding container 57d2f98e0f9e8e63c3e581402eec3c188883414d2114df2402087ecc3ab1a633: Status 404 returned error can't find the container with id 57d2f98e0f9e8e63c3e581402eec3c188883414d2114df2402087ecc3ab1a633
Sep 05 20:21:18.230554 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.230517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8" event={"ID":"46fa9287-f9fc-4f17-b7c4-84343a3d764f","Type":"ContainerStarted","Data":"e2ed072f860bf03b047661a9e37722dd8f50a553751e302a3764c4bdee33e789"}
Sep 05 20:21:18.230958 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.230573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8" event={"ID":"46fa9287-f9fc-4f17-b7c4-84343a3d764f","Type":"ContainerStarted","Data":"57d2f98e0f9e8e63c3e581402eec3c188883414d2114df2402087ecc3ab1a633"}
Sep 05 20:21:18.230958 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.230651    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:18.246233 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.246164    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8" podStartSLOduration=1.24615063 podStartE2EDuration="1.24615063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:21:18.245328056 +0000 UTC m=+2287.662401480" watchObservedRunningTime="2026-09-05 20:21:18.24615063 +0000 UTC m=+2287.663224053"
Sep 05 20:21:18.602286 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.602258    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wmxb8_905ab463-9ee2-45c9-ac7b-d3fd9f682065/dns/0.log"
Sep 05 20:21:18.621570 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.621533    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wmxb8_905ab463-9ee2-45c9-ac7b-d3fd9f682065/kube-rbac-proxy/0.log"
Sep 05 20:21:18.734330 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:18.734294    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h7bb2_c70e9649-272e-43b4-a83e-730d12d26234/dns-node-resolver/0.log"
Sep 05 20:21:19.286393 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:19.286364    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzr5g_9a9507e7-ca6e-41ff-b451-a22c805f69e9/node-ca/0.log"
Sep 05 20:21:20.757591 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:20.757559    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ss2lh_21de3aa5-6e75-48cb-9e65-358b8245f35f/discovery/0.log"
Sep 05 20:21:21.830309 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:21.830273    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzfw7_24af6bd4-d01c-4e94-91d2-4d01efde2be2/serve-healthcheck-canary/0.log"
Sep 05 20:21:22.401086 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:22.401060    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7d8x_5f2d3274-8edc-47da-b096-e65ecbb6a8e5/kube-rbac-proxy/0.log"
Sep 05 20:21:22.420623 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:22.420591    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7d8x_5f2d3274-8edc-47da-b096-e65ecbb6a8e5/exporter/0.log"
Sep 05 20:21:22.440980 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:22.440958    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7d8x_5f2d3274-8edc-47da-b096-e65ecbb6a8e5/extractor/0.log"
Sep 05 20:21:22.740035 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:22.739938    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chcpw_d6735bb9-81da-42b3-9293-9c88493ca967/konnectivity-agent/0.log" limit=1024
Sep 05 20:21:24.244698 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:24.244671    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-rh2v8"
Sep 05 20:21:24.859563 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:24.859534    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cjj8p_0cb46996-1381-4b32-9c28-49a2bf161710/server/0.log"
Sep 05 20:21:24.998079 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:24.998040    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xlm5x_1542feed-728f-463c-a706-3e4b5e82c240/manager/0.log"
Sep 05 20:21:25.201772 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:25.201688    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-qbtrn_4c7b57ab-b186-4fea-9a80-fb20df883193/manager/0.log"
Sep 05 20:21:25.218157 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:25.218125    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-qbtrn_4c7b57ab-b186-4fea-9a80-fb20df883193/copy-manifests/0.log"
Sep 05 20:21:26.633129 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:26.633095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6b9f469976-l2zbd_f2c9221f-d221-49ce-ad3d-aecb7ea01145/manager/0.log"
Sep 05 20:21:27.973512 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:27.973476    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ss2lh_21de3aa5-6e75-48cb-9e65-358b8245f35f/discovery/0.log"
Sep 05 20:21:30.270977 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:30.270949    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ss2lh_21de3aa5-6e75-48cb-9e65-358b8245f35f/discovery/0.log"
Sep 05 20:21:31.868842 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:31.868805    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-crmwc_2ea818ff-b0f7-41e2-a5c0-563fc484c7b6/migrator/0.log"
Sep 05 20:21:31.911090 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:31.911057    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-crmwc_2ea818ff-b0f7-41e2-a5c0-563fc484c7b6/graceful-termination/0.log"
Sep 05 20:21:33.315202 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.315161    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/kube-multus-additional-cni-plugins/0.log"
Sep 05 20:21:33.334832 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.334799    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/egress-router-binary-copy/0.log"
Sep 05 20:21:33.354928 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.354905    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/cni-plugins/0.log"
Sep 05 20:21:33.374369 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.374346    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/bond-cni-plugin/0.log"
Sep 05 20:21:33.394397 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.394379    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/routeoverride-cni/0.log"
Sep 05 20:21:33.414409 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.414377    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/whereabouts-cni-bincopy/0.log"
Sep 05 20:21:33.434297 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.434277    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zb8q_f58c4c6b-5ea4-42b0-b6bf-e57d5b97582f/whereabouts-cni/0.log"
Sep 05 20:21:33.885233 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.885200    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xhprs_0973d75e-376f-4828-a73a-9282fbbd8a9d/kube-multus/0.log"
Sep 05 20:21:33.950942 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.950913    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f82tc_40555d87-452a-46ac-a4ac-fba76c45fd18/network-metrics-daemon/0.log"
Sep 05 20:21:33.966411 ip-10-0-129-128 kubenswrapper[2575]: I0905 20:21:33.966384    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f82tc_40555d87-452a-46ac-a4ac-fba76c45fd18/kube-rbac-proxy/0.log"